Neural network coding and decoding method and system, and storage medium

Abstract

Systems and methods for encoding and decoding neural network data are provided. A method includes receiving a neural network representation (NNR) code stream including an NNR unit group (GON) representing an independent neural network having a topology, the NNR unit group (GON) including an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit; and reconstructing the independent neural network with the topology by decoding the NNR unit group (GON).

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 042,298 filed on June 22, 2020, U.S. Provisional Application No. 63 / 047,214 filed on July 1, 2020, U.S. Provisional Application No. 63 / 088,304 filed on October 6, 2020, U.S. Provisional Application No. 63 / 090,131 filed on October 9, 2020, and U.S. Application No. 17 / 324,623 filed on May 19, 2021, the disclosures of which are incorporated by reference into this application. Technical Field

[0002] Embodiments of the present disclosure relate to encoding and decoding of neural network data, and more particularly to parameter sets and header designs for compressed neural network representations. Background Art

[0003] Encoding and decoding of neural network data are provided. Summary of the invention

[0004] Embodiments of the present disclosure include techniques for parameter set and head design for compressed neural network representations.

[0005] Embodiments of the present disclosure include techniques for partial and independent unit design of compressed neural network representations.

[0006] Embodiments of the present disclosure include techniques for parameter set and clustering unit design for compressed neural network representations.

[0007] One or more embodiments include a decoding method of a neural network. The method includes: receiving a neural network representation NNR code stream including an NNR unit group GON, wherein the NNR unit group GON represents an independent neural network with a topology, wherein the NNR unit group GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit; and reconstructing the independent neural network with the topology by decoding the NNR unit group GON.

[0008] According to an embodiment, the group of NNR units GON is included in one or more aggregated NNR units of the neural network representation NNR code stream.

[0009] According to an embodiment, the group of NNR units GON is comprised in a single aggregated NNR unit.

[0010] According to an embodiment, the single aggregated NNR unit comprises a first syntax element indicating a type of the single aggregated NNR unit as a self-contained NNR aggregation unit.

[0011] According to an embodiment, the first syntax element is included in a header of the single aggregated NNR unit.

[0012] According to an embodiment, the NNR model parameter set unit comprises a second syntax element indicating that the NNR unit referring to the NNR model parameter set unit is independently decodable.

[0013] According to an embodiment, the second syntax element is included in a header of the NNR model parameter set unit.

[0014] According to an embodiment, the NNR layer parameter set unit includes a third syntax element, the third syntax element indicating that the NNR unit referring to the NNR layer parameter set unit is independently decodable

[0015] The third syntax element is included in a header of the NNR layer parameter set unit.

[0016] In the NNR unit group GON, the NNR model parameter set unit is followed by the NNR layer parameter set unit.

[0017] According to one or more embodiments, a neural network decoding system is provided. The system includes at least one processor, a memory, the memory stores computer code, and the computer code is configured to enable the at least one processor to receive a neural network representation NNR code stream including an NNR unit group GON, the NNR unit group GON represents an independent neural network with a topology, and the NNR unit group GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit; and reconstruct the independent neural network with the topology by decoding the NNR unit group GON.

[0018] According to an embodiment, the group of NNR units GON is included in one or more aggregated NNR units of the neural network representation NNR code stream.

[0019] According to an embodiment, said neural network representation GON is comprised in a single aggregated NNR unit.

[0020] The single aggregated NNR unit includes a first syntax element indicating a type of the single aggregated NNR unit as a self-contained NNR aggregation unit and causing the at least one processor to determine that the single aggregated NNR unit is self-contained based on the first syntax element.

[0021] According to an embodiment, the first syntax element is included in a header of the single aggregated NNR unit.

[0022] According to an embodiment, the NNR model parameter set unit includes a second syntax element, which indicates that the NNR unit referring to the NNR model parameter set unit is independently decodable, and causes the at least one processor to determine that the NNR unit referring to the NNR model parameter set unit is independently decodable based on the second syntax element.

[0023] According to an embodiment, the second syntax element is included in a header of the NNR model parameter set unit.

[0024] According to an embodiment, the NNR layer parameter set unit includes a third syntax element, which indicates that the NNR unit referring to the NNR layer parameter set unit is independently decodable, and enables the at least one processor to determine that the NNR unit referring to the NNR layer parameter set unit is independently decodable based on the third syntax element.

[0025] According to an embodiment, the third syntax element is included in a header of the NNR layer parameter set unit.

[0026] According to one or more embodiments, a non-volatile computer-readable medium storing computer instructions is provided, and at least one processor receives a neural network representation NNR code stream including an NNR unit group GON, and the NNR unit group GON represents an independent neural network with a topology. When the computer instructions are executed by the at least one processor, the at least one processor causes the at least one processor to: reconstruct the independent neural network with the topology by decoding the NNR unit group GON, wherein the NNR unit group GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0028] Figure 1 is an illustration of an environment in which the methods, apparatuses, and systems described in this application may be implemented according to an embodiment.

[0029] Figure 2 yes Figure 1 A block diagram of exemplary components of one or more devices.

[0030] Figure 3 Schematic diagram of an NNR code stream according to an embodiment.

[0031] Figure 4 is a schematic diagram of an NNR unit according to an embodiment.

[0032] Figure 5 is a schematic diagram of a clustered NNR unit according to an embodiment.

[0033] Figure 6 is a schematic diagram of a group of NNR units according to an embodiment.

[0034] Figure 7 is a block diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0035] Figure 1 1 is a diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented. Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0036] User device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 120. For example, user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some embodiments, user device 110 can receive information from platform 120 and / or send information to platform 120.

[0037] The platform 120 includes one or more devices as described elsewhere herein. In some embodiments, the platform 120 may include a cloud server or a cloud server group. In some embodiments, the platform 120 may be designed to be modular so that software components can be swapped in or out according to specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different purposes.

[0038] In some embodiments, as shown, the platform 120 may be hosted in a cloud computing environment 122. It is worth noting that while the embodiments described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some embodiments, the platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0039] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of one or more systems and / or one or more devices used to host the aforementioned platform 120. As shown, cloud computing environment 122 can include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually referred to as "computing resource 124").

[0040] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some embodiments, the computing resources 124 can host the above-mentioned platform 120. Cloud resources can include computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some embodiments, the computing resources 124 can communicate with other computing resources 124 through wired connections, wireless connections, or a combination of wired and wireless connections.

[0041] like Figure 1 As further shown, the computing resources 124 include a set of cloud resources, such as one or more applications ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, and the like.

[0042] Applications 124-1 include one or more software applications that can be provided to or accessed by user device 110 and / or platform 120. Applications 124-1 can eliminate the need to install and execute software applications on user device 110. For example, applications 124-1 can include software associated with platform 120 and / or any other software that can be provided through cloud computing environment 122. In some embodiments, one application 124-1 can send / receive information to / from one or more other applications 124-1 through virtual machine 124-2.

[0043] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Depending on the use of virtual machine 124-2 and the degree of correspondence with any real machine, virtual machine 124-2 can be a system virtual machine or a process virtual machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some embodiments, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-duration data transfers.

[0044] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage system or device of computing resource 124. In some embodiments, within the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage, so that the storage system may be accessed without regard to physical storage or heterogeneous structures. This separation may allow administrators of storage systems to have flexibility in how to manage storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This may optimize the performance of storage usage, server consolidation, and / or non-destructive file migration.

[0045] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute concurrently on a host, such as computing resource 124. Hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0046] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0047] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different Figure 1 The devices and / or networks shown are arranged differently. Figure 1 Two or more of the devices shown may be implemented in a single device, or the Figure 1 The single device shown is implemented as multiple distributed devices. Additionally or alternatively, a set of devices (eg, one or more devices) of environment 100 may perform one or more functions that are described as being performed by another set of devices of environment 100.

[0048] Figure 2 yes Figure 1 Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0049] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic storage, and / or optical storage), which is used to store information and / or instructions for use by the processor 220.

[0050] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium, and a corresponding drive.

[0051] Input components 250 include components that allow device 200 to receive information through, for example, user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, and / or actuators). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0052] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices through, for example, a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0053] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by non-volatile computer-readable media such as memory 230 and / or storage component 240. Computer-readable media is defined herein as non-volatile storage devices. Storage devices include storage space within a single physical storage device or storage space distributed on multiple physical storage devices.

[0054] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device through the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardware circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuitry and software.

[0055] Figure 2 The number and arrangement of components shown are provided as examples. In practice, device 200 may include additional components, fewer components, different components, or different components. Figure 2 Additionally or alternatively, one set of components (eg, one or more components) of device 200 may perform one or more functions that are described as being performed by another set of components of device 200.

[0056] The embodiments described below may be implemented by one or more components of environment 100. For example, the embodiments described below may be implemented by user device 110 or platform 120. The embodiments described below may be implemented by at least one processor and a memory storing computer instructions. The computer instructions may be configured to perform the functions of the embodiments when executed by at least one processor. For example, the computer instructions may be configured to cause at least one processor to implement at least one NNR code stream as described below, including sending, receiving, and decoding the NNR code stream.

[0057] Embodiments of the present disclosure may include the following improvements to parameter sets and headers in NNR HLS:

[0058] Aspect 1: An embodiment may add the definition of a "group of NNR units (GON)", which is a group of NNR units including a whole or part of a neural network with a topology. Each GON is self-contained and independently decodable. In an NNR code stream, there may be one or more GONs. Each GON starts with an NNR model parameter set unit, followed by an NNR layer parameter set, an NNR topology unit, and other NNR units. The concept of a GON may be equivalent to a coded video sequence (CVS, coded video sequence) in VVC / HEVC / AVC, except for the concept of timing. A GON may not have a sequence order with counting information. Each GON may be contained in one or more aggregated NNR units. When an aggregated NNR unit represents a part or all of a neural network with a topology, each aggregated NNR unit may include at least one associated model parameter set and one associated layer parameter set. Then, the scope of the associated model parameter set and the associated layer parameter set may be limited to the NNR units contained in (or associated with) the aggregated NNR unit. In this case, the aggregated NNR unit may have one or more identifiers to indicate one or more reference parameters.

[0059] Aspect 2: An embodiment may include syntax elements of model_parameter_set_id and layer_parameter_set_id, and define reference rules for NNR model parameter sets and layer parameter sets and their ranges, so that all layer parameter sets in an NNR unit group refer to only one model parameter set, and one or more NNR units refer to the layer parameter set. The concepts of NNR model parameter sets and layer parameter sets may be equivalent to sequence parameter sets and picture parameter sets in VVC / HEVC / AVC, respectively. Each aggregated NNR unit may have its own parameter set ID to specify which parameter sets are associated with the NNR units in the aggregated NNR unit.

[0060] Aspect 3: An embodiment may include the syntax element sparsification_flag in the model parameter set and the layer parameter set as model_sparsification_present_flag and layer_sparsification_present_flag, respectively, and impose a constraint that the value of layer_sparsification_present_flag shall be equal to 0 when model_sparsification_present_flag is equal to 1, to avoid the presence of the same parameter in both parameter sets. In an embodiment, when a sparsification parameter is present in both the model parameter set and the layer parameter set, the value of the sparsification parameter overrides the value of the sparsification parameter in the layer parameter set.

[0061] Aspect 4: An embodiment may include a syntax element quantization_method_flags in the model parameter set and the layer parameter set as model_quantization_method_flags and layer_quantization_method_flags, respectively, and impose a constraint that may require model_quantization_method_flags & layer_quantization_method_flags to be equal to 0 to avoid signaling the syntax element in both the model parameter set and the layer parameter set. In an embodiment, when a quantization parameter is present in both the model parameter set and the layer parameter set, the value of the quantization parameter overrides the value of the quantization parameter in the layer parameter set.

[0062] Aspect 5: An embodiment may include general_profile_idc, which indicates the profile into the NNR start unit (header). The scope of general_profile_idc in the NNR start unit may be the entire code stream.

[0063] Aspect 6: An embodiment may not include nnr_model_parameter_set_unit_header(), nnr_layer_parameter_set_unit_header(), because it may not have a use case where parameter sets and start units have their headers.

[0064] The above aspects and other aspects of the embodiments of the present disclosure can be understood based on the following description.

[0065] [Aspect 1]

[0066] Embodiments of the present disclosure may include the definition of a "group of NNR units (GON)", which is a group of NNR units including a whole or part of a neural network with a topology. Each GON is self-contained and independently decodable. In an NNR codestream, there may be one or more GONs. This means that the NNR codestream may contain multiple neural networks with multiple topologies. The concept of GON may be equivalent to a coded video sequence (CVS) in VVC / HEVC / AVC, except for the concept of timing. GON may not have a sequence order with counting information. After the NNR start unit exists in each NNR codestream, each GON may start with an NNR model parameter set unit, followed by an NNR layer parameter set, an NNR topology unit, and other NNR units.

[0067] Embodiments of the present disclosure may, for example, send, receive and / or decode NNR code streams. The following describes NNR code streams and their syntax elements that may be implemented by embodiments of the present disclosure.

[0068] (A) Description of NNR codestream and general codestream syntax elements

[0069] (1) NNR bitstream

[0070] refer to Figure 3 , the NNR code stream 300 is composed of a sequence of NNR units 302. It may be required that the first NNR unit in the NNR units 302 in the NNR code stream 300 is an NNR start unit (eg, an NNR unit of the NNR_STR type).

[0071] (2) NNR unit

[0072] refer to Figure 4 , one of the multiple NNR units in NNR unit 302 may be NNR unit 310. NNR unit 310 is a data structure for carrying neural network data and related metadata, which is compressed or represented according to an embodiment of the present disclosure.

[0073] The NNR unit 310 carries compressed or uncompressed information about neural network metadata, topology information, complete or partial layer data, filters, kernels, biases, quantized weights, tensors, etc.

[0074] The NNR unit 310 may include or consist of the following data elements: (a) NNR unit size 312: This data element signals the total byte size of the NNR unit 310, including the NNR unit size. (b) NNR Unit Header 314: This data element contains information about the NNR unit type and related metadata. (c) NNR Unit Payload 316: This data element contains compressed or uncompressed data related to the neural network.

[0075] (3) Aggregation of NNR units

[0076] refer to Figure 5 , one of the multiple NNR units in the NNR unit 302 may be an aggregated NNR unit 320. The aggregated NNR unit 320 is an NNR unit that carries multiple NNR units 310 in its payload. The aggregated NNR unit 320 provides a grouping mechanism for several NNR units 310 that are related to each other and benefit from aggregation under a single NNR unit 302. For example, the aggregated NNR unit 310 may include or consist of the following data elements: (a) Aggregate NNR Unit Size 322: This data element signals the total byte size of the aggregate NNR unit 310, including the NNR unit size. (b) Aggregate NNR Unit Header 324: This data element contains information about the Aggregate NNR Unit type and related metadata. (c) Aggregate NNR unit payload 326: This data element contains multiple NNR units 310.

[0077] (B) Group of NNR Units (GON)

[0078] The NNR group of units (GON) 330 includes an independent neural network having a topology. The GON 330 includes or consists of an NNR model parameter set unit 331, one or more NNR layer parameter set units 332, an NNR topology unit 333, one or more NNR quantization units 334, and an NNR compressed data unit 335. After the NNR start unit exists in the NNR code stream, the GON 330 starts with the NNR model parameter set unit 331, followed by the NNR layer parameter set unit 332, the NNR topology unit 333, and other units.

[0079] The NNR code stream 300 includes one or more GONs. Each GON is independently decodable and extractable.

[0080] Each GON 330 may be included in one or more aggregated NNR units 320. For example, each unit of the GON 330 may be a corresponding NNR unit 310 from the NNR units 310 of a single aggregated NNR unit 320 or a plurality of aggregated NNR units 320. When the aggregated NNR unit 320 represents a partial or entire neural network having a topology, each aggregated NNR unit 320 may include at least one associated model parameter set and one associated layer parameter set. Then, the scope of the associated model parameter set and the associated layer parameter set may be limited to the NNR units included in (or associated with) the aggregated NNR unit 320. In this case, the aggregated NNR unit 320 may have one or more identifiers to indicate one or more reference parameters.

[0081] (C) Aggregate NNR unit head semantics

[0082] The aggregated NNR unit header 324 may include syntax elements such as, but not limited to, nnr_aggregate_unit_type, entry_points_present_flag, num_of_nnr_units_minus2, nnr_unit_type[i], nnr_unit_entry_point[i], quant_bitdepth[i], and ctu_scan_order[i].

[0083] The syntax element nnr_aggregate_unit_type indicates the type of the aggregated NNR unit 320 .

[0084] For example, the NNR aggregation unit types may be specified as shown in Table 1 below. Table 1: NNR aggregation unit types

[0085] The syntax element entry_points_present_flag indicates whether there are separate NNR unit entry points.

[0086] The syntax element num_of_nnr_units_minus2+2 indicates the number of NNR units 310 present in the payload of the NNR aggregation unit.

[0087] The syntax element nnr_unit_type[i] indicates the NNR unit type of the NNR unit 310 with index i. This value may be required to be the same as the NNR unit type of the NNR unit 310 at index i.

[0088] The syntax element nnr_unit_entry_point[i] indicates the byte offset from the start of the NNR aggregation unit 320 to the start of the NNR unit 310 in the payload of the NNR aggregation unit and at index i. This value may be required to be not equal to or greater than the total byte size of the NNR aggregation unit 320. The syntax element nnr_unit_entry_point value may be used for fast and random access of the NNR unit 310 within the NNR aggregation unit payload.

[0089] The syntax element quant_bitdepth[i] indicates the maximum bit depth of the quantized coefficients of each tensor in the NNR aggregation unit 320 .

[0090] The syntax element ctu_scan_order[i] indicates the CTU-wise scan order of each tensor in the NNR aggregation unit 320. A value of 0 indicates that the CTU-wise scan order is a raster scan order in the horizontal direction, and a value of 1 indicates that the CTU-wise scan order is a raster scan order in the vertical direction.

[0091] [Aspect 2]

[0092] Embodiments of the present disclosure may include model_parameter_set_id and layer_parameter_set_id, and define reference rules for NNR model parameter sets and layer parameter sets and their ranges, so that all layer parameter sets in GON refer to only one model parameter set, and one or more NNR quantization and compression data units refer to the layer parameter set. Therefore, the structure of the hierarchical reference is that each NNR quantization and compression data unit refers to an NNR layer parameter set unit, which refers to an NNR model parameter set unit.

[0093] The concepts of NNR model parameter set and layer parameter set can be equivalent to the sequence parameter set and picture parameter set in VVC / HEVC / AVC, respectively.

[0094] Embodiments of the present disclosure may implement the following constraints: (1) The NNR code stream may start with an NNR start unit (NNR_STR). (2) There may be a single NNR model parameter set (NNR_MPS) in the NNR codestream 300 , which will precede any NNR_NDU in the NNR codestream 300 . (3) The NNR layer parameter set will be active until the next NNR layer parameter set in the NNR codestream 300 or until the boundary of the aggregated NNR unit 320 is reached. (4) The syntax element ref_id value is required to be unique in the NNR codestream 300. (5) NNR_TPL or NNR_QNT unit; if present in the NNR codestream 300, it should precede any NNR_NDU that references its data structure (e.g., ref_id).

[0095] Embodiments of the present disclosure may implement an NNR model parameter set that may be present in the NNR codestream 300. An example syntax of the NNR model parameter set is provided in Table 2 below. Table 2: NNR model parameter set payload syntax

[0096] The syntax element mps_model_parameter_set_id indicates the identifier of the model parameter set referenced by other syntax elements.

[0097] Embodiments of the present disclosure may implement an NNR layer parameter set present in the NNR codestream 300. An example syntax of the NNR layer parameter set is provided in Table 3 below. Table 3: NNR layer parameter set element payload syntax

[0098] The syntax element lps_layer_parameter_set_id indicates the identifier of the layer parameter set referenced by other syntax elements.

[0099] The syntax element lps_model_parameter_set_id indicates the value of mps_model_parameter_set_id of the model parameter set referenced by the layer parameter set. The value of lps_model_parameter_set_id may be required to be the same in all layer parameter sets in a GON.

[0100] Embodiments of the present disclosure may implement an NNR quantization unit header of an NNR quantization unit present in the NNR code stream 300. An example syntax of the NNR quantization unit header is provided in Table 4 below. Table 4: NNR quantization unit header syntax

[0101] The syntax element qu_layer_parameter_set_id indicates the value of lps_layer_parameter_set_id of the layer parameter set referenced by the NNR quantization unit.

[0102] Embodiments of the present disclosure may implement an NNR compressed data unit header for an NNR compressed data unit present in the NNR codestream 300. An example syntax of the NNR compressed data unit header is provided in Table 5 below. Table 5: NNR Compressed Data Unit Header Syntax

[0103] The syntax element cdu_layer_parameter_set_id indicates the value of lps_layer_parameter_set_id of the layer parameter set referenced by the NNR compressed data unit.

[0104] [Aspect 3]

[0105] Embodiments of the present disclosure may include a syntax element sparsification_flag in an NNR model parameter set unit and an NNR layer parameter set unit as model_sparsification_present_flag and layer_sparsification_present_flag, respectively, and impose a constraint that the value of layer_sparsification_present_flag should be equal to 0 when model_sparsification_present_flag is equal to 1, to avoid the presence of the same parameters in the two parameter sets. In an embodiment, for global indication of sparsification_performance_map() in a GON, it is required to set model_sparsification_present_flag equal to 1. Otherwise, for local indication, model_sparsification_present_flag is set equal to 0 and layer_sparsification_present_flag is set equal to 1. Signaling sparsification_performance_map() in the NNR model parameter set and layer parameter set units may not be efficient and may cause confusion.

[0106] In an embodiment, when a sparsification parameter is present in both the model parameter set and the layer parameter set, the value of the sparsification parameter overrides the value of the sparsification parameter in the layer parameter set.

[0107] An example syntax for an NNR model parameter set is provided in Table 6 below. Table 6: NNR model parameter set payload syntax

[0108] An example syntax for an NNR layer parameter set is provided in Table 7 below. Table 7: NNR layer parameter set element payload syntax

[0109] When model_sparsification_flag is equal to 1, the value of layer_sparsification_flag is required to be equal to 0.

[0110] [Aspect 4]

[0111] An embodiment may include model_quantization_method_present_flag and layer_quantization_method_present_flag in the model parameter set and the layer parameter set, respectively, and impose a constraint that the value of layer_quantization_method_present_flag should be equal to 0 when model_quantization_method_present_flag is equal to 1 to avoid the same parameters in the two parameter sets.

[0112] An example syntax for an NNR model parameter set is provided in Table 8 below. Table 8: NNR model parameter set payload syntax

[0113] The syntax element model_quantization_method_flags specifies one or more quantization methods for the entire model. If multiple models are specified, they are combined by "or". As an example, the quantization methods shown in Table 9 can be specified. Table 9: Quantification methods Quantitative methods Quantization method ID value Scalar uniform NNR_QSU 0x01 Codebook NNR_QCB 0x02 Reserved 0x03-0xFF

[0114] An example syntax for an NNR layer parameter set is provided in Table 10 below. Table 10: NNR layer parameter set element payload syntax

[0115] The syntax element layer_quantization_method_flags specifies the quantization method used for the data contained in the NNR compressed data unit referenced by this layer parameter set. If multiple models are specified, they are combined by "or". As an example, the quantization methods shown in the following Table 11 can be specified. Table 11: Quantification methods Quantitative methods Quantization method ID value Scalar uniform NNR_QSU 0x01 Codebook NNR_QCB 0x02 Reserved 0x03-0xFF

[0116] The syntax elements model_quantization_method_flags&quantization_method_flags are required to be equal to 0.

[0117] In an embodiment, when the quantization parameter exists in both the model parameter set and the layer parameter set, the value of the quantization parameter overrides the value of the quantization parameter in the layer parameter set. Such an embodiment is described below with reference to Tables 12 to 16.

[0118] An example syntax for an NNR model parameter set according to such an embodiment is provided in Table 12 below. Table 12: NNR model parameter set payload syntax

[0119] The syntax element topology_carriage_flag indicates whether the NNR codestream 300 carries the topology internally or externally. When set to 1, it indicates that the topology is carried in one or more NNR unit types "NNR_TPL". If it is 0, it indicates that the topology is provided externally (i.e., out-of-band relative to the NNR codestream 300).

[0120] The syntax element mps_quantization_method_flag indicates the quantization method used for the model in the NNR compressed data unit that refers to this model parameter set. If multiple models are specified, they are combined by "or". In such an embodiment, as an example, the quantization methods shown in Table 13 can be specified. Table 13: Quantification methods Quantitative methods Quantization method ID value Scalar uniformity NNR_QSU 0x01 Codebook NNR_QCB 0x02 reserve 0x03-0xFF

[0121] The syntax element mps_qp_density indicates density information of the syntax element mps_quantization_parameter in the NNR compressed data unit referring to the model parameter set.

[0122] The syntax element mps_quantization_parameter indicates a quantization parameter of a scalar uniform quantization of a parameter of each layer of a neural network used for arithmetic coding in an NNR compressed data unit referring to the model parameter set.

[0123] The syntax element mps_sparsification_flag indicates whether sparsification is applied to the model in the NNR compressed data units that refer to this model parameter set.

[0124] The syntax element sparsification_performance_map() indicates the mapping between different sparsification thresholds and the resulting NN inference accuracy. The resulting accuracy can be provided separately for different aspects or characteristics of the NN's output. For a classifier NN, each sparsification threshold can be mapped to a separate accuracy for each class in addition to considering the total accuracy for all classes. The classes can be ordered based on the neural network output order (e.g., the order specified during training).

[0125] The syntax element count_thresholds indicates the number of thinning thresholds.

[0126] The syntax element sparsification_threshold indicates a threshold applied to the weights of the decoded neural network in order to set the weights to zero (eg, weights whose values ​​are less than the threshold are set to zero).

[0127] The syntax element non_zero_ratio indicates the non-zero ratio achieved by applying sparsification_threshold to sparsify the weights.

[0128] The syntax element nn_accuracy indicates the overall accuracy of the NN (eg, by considering the classification accuracy of all classes).

[0129] The syntax element count_classes indicates the number of classes that provide separate precision for each sparsification threshold.

[0130] The syntax element nn_class_accuracy indicates the accuracy for a certain class when a certain sparsification threshold is applied.

[0131] The syntax element ctu_partition_flag indicates whether block partitioning is enabled for the weight tensor of each layer of the neural network. A value of 0 indicates that block partitioning is disabled, and a value of 1 indicates that block partitioning is enabled, where the partitioned blocks are defined as coding tree units (CTUs).

[0132] The syntax element max_ctu_dim_flag indicates the model-wise max CTU dimension of the weight tensor of the neural network: gctu_dim = (64 >> max_ctu_dim_flag). The tensor-wise max CTU width is scaled by the kernel size of each convolution tensor: max_ctu_height = gctu_dim, max_ctu_width = gctu_dim*kernel_size. The height / width of the right / bottom CTU can be smaller than max_ctu_height / max_ctu_width.

[0133] An example syntax for the NNR layer parameter set is provided in Table 14 below. Table 14: NNR layer parameter set element payload syntax

[0134] The syntax element lps_quantization_method_flags specifies the quantization method used for data contained in the NNR compressed data unit that references this layer parameter set. If multiple models are specified, they are combined by "or". As an example, the quantization methods shown in the following Table 15 can be specified. Table 15: Quantification methods

[0135] The syntax element lps_quantization_step_size indicates the step size interval of the scalar uniform quantization method used for the data contained in the NNR compressed data unit that references this layer parameter set.

[0136] The syntax element quantization_map() indicates a codebook used for a codebook-based quantization method.

[0137] The syntax element compressed_flag indicates whether the quantization map data is further compressed.

[0138] The syntax element compression_format is an enumerated list that takes one of the values ​​shown in Table 16 below and indicates the compression method used to further compress quantization_map_data(): Table 16: Compression formats and values

[0139] The syntax element index_precision indicates the integer type used in the key value of quantization_map_data(). A value of 0 indicates 8-bit precision, a value of 1 indicates 16-bit precision, and a value of 2 indicates 32-bit precision. Other values ​​are reserved.

[0140] The syntax element quantization_map_data() indicates {[index<index_precision> :value<float(32)> ]}: where the index is a quantization value indicator and the second value is a signed floating point value corresponding to the quantization value index.

[0141] The syntax element lps_sparsification_flag indicates whether sparsification is applied to the models in the NNR compressed data units that refer to this layer parameter set.

[0142] The syntax element sparsification_performance_map() indicates the mapping between different sparsification thresholds and the resulting NN inference accuracy. The resulting accuracy can be provided separately for different aspects or characteristics of the NN's output. For a classifier NN, each sparsification threshold can be mapped to a separate accuracy for each class in addition to considering the total accuracy for all classes. The classes can be ordered based on the neural network output order (i.e. the order specified during training).

[0143] When mps_sparsification_flag&lps_sparsification_flag is equal to 1, the values ​​of the syntax elements in the sparsification_performance_map() in the layer parameter set may be used in the NNR compressed data unit that references the layer parameter set by overwriting the values ​​of the syntax elements in the sparsification_performance_map() in the model parameter set.

[0144] [Aspect 5]

[0145] Embodiments of the present disclosure may include general_profile_idc indicating a profile in the NNR start unit (header). The profile information is important for providing information about what encoding algorithm and topology is used in the NNR code stream 300.

[0146] Embodiments of the present disclosure may implement an NNR start unit header present in the NNR code stream 300. An example syntax of the NNR start unit header is provided in Table 17 below. Table 17: NNR start unit header syntax nnr_start_unit_header(){ Descriptors general_profile_idc u(8) }

[0147] The syntax element general_profile_idc indicates the profile to which the NNR codestream 300 conforms. The codestream may be required to not contain values ​​of general_profile_idc other than those specified in the specification.

[0148] [Aspect 6]

[0149] An embodiment of the present disclosure may not include nnr_model_parameter_set_unit_header(), nnr_layer_parameter_set_unit_header() because it may not have a use case where parameter sets and start units have their headers.

[0150] [Additional aspects]

[0151] Additional aspects of embodiments of the disclosure are described below.

[0152] (A) When the value of partial_data_counter in the NNR unit header is equal to 0, the value of independently_decodable_flag in the NNR unit header may be equal to 0, because when the NNR unit contains the entire data of the topology, the unit should be independently decodable. According to an embodiment, in this case, the value of independently_decodable_flag in the NNR unit header may be determined by the decoder to be equal to 0.

[0153] An example syntax for the NNR unit is provided in Table 18 below. Table 18: NNR unit syntax

[0154] An example syntax for the NNR unit size is provided in Table 19 below. Table 19: NNR unit size syntax

[0155] An example syntax for the NNR unit header is provided in Table 20 below. Table 20: NNR unit header syntax

[0156] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of this NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates no partial information (i.e., the data in this NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of partial information (i.e., the data in this NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backward to indicate the total number of initial partitions.

[0157] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to achieve successful decodability / decompressibility. When the value of partial_data_counter is equal to 0, the value of independently_decodable_flag is required to be equal to 0.

[0158] (B) When the NNR unit is independently decodable, partial_data_counter may not be signaled in the nnr_unit_header, and the value of partial_data_counter may be inferred (eg, by a decoder) to be equal to 0.

[0159] An example syntax for the NNR unit header is provided in Table 21 below. Table 21: NNR unit header syntax

[0160] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to achieve successful decodability / decompressibility.

[0161] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of the partial information (i.e., the data in the NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backwards to indicate the total number of initial partitions. When not present, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0.

[0162] (C) When the NNR unit is independently decodable, the value of partial_data_counter may be equal to 0.

[0163] An example syntax for the NNR unit header is provided in Table 22 below. Table 22: NNR unit header syntax

[0164] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to achieve successful decodability / decompressibility.

[0165] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of this NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates that there is no partial information (i.e., the data in this NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of partial information (i.e., the data in this NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backward to indicate the total number of initial partitions. When independently_decodable_flag is equal to 1, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0.

[0166] (D) According to an embodiment, the semantics of independently_decodable_flag may be changed as follows.

[0167] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit and no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and it is complete). A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to obtain successful decodability / decompressibility.

[0168] (E) According to an embodiment, the semantics of partial_data_counter may be changed as follows.

[0169] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and it is complete), and the NNR unit with partial_data_counter equal to 0 is independently decodable. A value greater than 0 indicates the index of partial information (i.e., the data in the NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backwards to indicate the total number of initial partitions. When independently_decodable_flag is equal to 1, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0.

[0170] (F) According to an embodiment, when nnr_unit_type is equal to NNR_STR or NNR_MPS, the value of independently_decodable_flag may be equal to 0.

[0171] (G) According to an embodiment, when nnr_unit_type is equal to NNR_STR, NNR_MPS, NNR_LPS, NNR_TPL, NNR_QNT, NNR_NDU, or NNR_AGG, the value of independently_decodable_flag may be equal to 0.

[0172] (H) According to an embodiment, when nnr_unit_type is equal to NNR_STR or NNR_AGG, the value of partial_data_counter may be equal to 0.

[0173] (I) According to an embodiment, when nnr_unit_type is equal to NNR_AGG and nnr_aggregate_unit_type is equal to NNR_AGG_SLF, the value of partial_data_counter may be equal to 0.

[0174] (J) According to an embodiment, when nnr_unit_type is equal to NNR_AGG and nnr_aggregate_unit_type is equal to NNR_AGG_SLF, the value of independently_decodable_flag may be equal to 0.

[0175] (K) According to an embodiment, lps_independently_decodable_flag is signaled in the layer parameter set and its semantics is as shown in Table 23 below. Table 23: NNR layer parameter set syntax

[0176] The syntax element lps_independently_decodable_flag indicates whether the NNR unit referencing this layer parameter set is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit referencing the layer parameter set is not independently decodable and the payload of the NNR unit should be combined with NNR units referencing other layer parameter sets to obtain successful decodability / decompressibility.

[0177] (L) According to an embodiment, the value of lps_independently_decodable_flag may be equal to the value of independently_decodable_flag in the NNR unit header.

[0178] (M) According to an embodiment, independently_decodable_flag is replaced by reserved bits as shown in the following Table 24. Table 24: NNR layer parameter set syntax

[0179] (N) According to an embodiment, syntax and semantics are presented in the NNR model parameter set as described below and as shown in Table 25. Table 25: NNR model parameter set unit header syntax

[0180] The syntax element mps_independently_decodable_flag equal to 0 indicates that the NNR unit of the reference model parameter set is independently decodable and the full or partial NN model should be successfully reconstructed using the NNR unit. A value of 1 indicates that the NNR unit of the reference model parameter set is not independently decodable and the payload of the NNR unit should be combined with NNR units that reference other model parameter sets to successfully construct a full or partial NN model.

[0181] (O) According to an embodiment, as described below and as shown in Table 26, the following syntax and semantics are presented in the NNR layer parameter set. Table 26: NNR layer parameter set unit header syntax

[0182] The syntax element lps_independently_decodable_flag equal to 0 indicates that the NNR unit referring to the layer parameter set is independently decodable and the full or partial NN model should be successfully reconstructed using the NNR unit. A value of 1 indicates that the NNR unit referring to the layer parameter set is not independently decodable and the payload of the NNR unit should be combined with NNR units referring to other layer parameter sets to successfully construct a full or partial NN model.

[0183] (P) According to an embodiment, the syntax structure shown in the following Table 27 may be used for the NNR unit header. Table 27: NNR unit head

[0184] The syntax element nnr_unit_type indicates the type of the NNR unit as specified in Table 28 below. Table 28: NNR unit types

[0185] The syntax element nnr_unit_header_extension_flag equal to 1 indicates that the syntax elements partial_data_counter and independently_decodable_flag are present in the NNR unit header. The syntax element nnr_unit_header_extension_flag equal to 0 indicates that the syntax elements partial_data_counter and independently_decodable_flag are not present in the NNR unit header.

[0186] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of the partial information (i.e., the data in the NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backward to indicate the total number of initial partitions. When not present, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0. The value of partial_data_counter can be in the range of 0 to 126 (including the end value). The value 127 is reserved for future proofing.

[0187] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to obtain successful decodability / decompressibility. When not present, the value of independently_decodable_flag can be inferred (e.g., by a decoder) to be equal to 0.

[0188] (Q) According to an embodiment, the syntax structure shown in the following Table 29 may be used for the NNR unit header. Table 29: NNR unit header syntax

[0189] The syntax element nnr_unit_type indicates the type of the NNR unit as specified in Table 30 below. Table 30: NNR unit types

[0190] The syntax element nnr_unit_header_extension_flag equal to 1 indicates that the syntax elements partial_data_counter and independently_decodable_flag are present in the NNR unit header. The syntax element nnr_unit_header_extension_flag equal to 0 indicates that the syntax elements partial_data_counter and independently_decodable_flag are not present in the NNR unit header.

[0191] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a particular topology element. A value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of the partial information (i.e., the data in the NNR unit should be concatenated with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backward to indicate the total number of initial partitions. When not present, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0. The value of partial_data_counter can be in the range of 0 to 127 (including the end value).

[0192] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to obtain successful decodability / decompressibility. When not present, the value of independently_decodable_flag can be inferred (e.g., by a decoder) to be equal to 0.

[0193] The syntax element reserved_bit may be used for future proofing.

[0194] (R) According to an embodiment, the syntax structure shown in the following Table 31 can be used for the NNR unit header. Table 31: NNR unit header syntax

[0195] The syntax element nnr_unit_type indicates the type of the NNR unit as specified in Table 32 below. Table 32: NNR unit types

[0196] The syntax element independent_self_contained_flag equal to 1 indicates that the NNR unit is independently decodable and self-contained (i.e., the data in this NNR unit is all the data associated with the topology element and it is complete). When the value of independent_self_contained_flag is equal to 1, the syntax elements partial_data_counter and independently_decodable_flag are not present in the NNR unit header, and the values ​​of partial_data_counter and independently_decodable_flag may be inferred (e.g., by the encoder) to be equal to 0. independent_self_contained_flag equal to 0 indicates that the syntax elements partial_data_counter and independent_decodable_flag are present in the NNR unit header. When independent_self_contained_flag is equal to 0, one of partial_data_counter and independently_decodable_flag may be required to have a non-zero value.

[0197] The syntax element partial_data_counter indicates the index of the partial data carried in the payload of the NNR data unit relative to the entire data of a specific topology element. A value of 0 indicates that there is no partial information (i.e., the data in the NNR unit is all the data associated with the topology element and is complete), and a value greater than 0 indicates the index of the partial information (i.e., the data in the NNR unit should be cascaded with the data in the accompanying NNR unit until the partial_data_counter of the NNR unit reaches 0). The counter can count backwards to indicate the total number of initial partitions. When not present, the value of partial_data_counter can be inferred (e.g., by a decoder) to be equal to 0. The value of partial_data_counter can be in the range of 0 to 126 (including the end value). The value 127 can be reserved for future proofing.

[0198] The syntax element independently_decodable_flag indicates whether the compressed data unit is independently decodable. A value of 0 indicates an independently decodable NNR unit. A value of 1 indicates that the NNR unit is not independently decodable and the payload of the NNR unit should be combined with other NNR units to obtain successful decodability / decompressibility. When not present, the value of independently_decodable_flag can be inferred (e.g., by a decoder) to be equal to 0.

[0199] (S) According to an embodiment, the syntax and semantics described below and shown in the following Table 33 are presented in the NNR model parameter set. Table 33: NNR model parameter set unit header syntax

[0200] The syntax element mps_independent_self_contained_flag equal to 0 indicates that the NNR unit of the reference model parameter set is independently decodable and it should be possible to successfully reconstruct a full or partial NN model using the NNR unit. A value of 1 indicates that the NNR unit of the reference model parameter set is not independently decodable and the payload of the NNR unit should be combined with NNR units that reference other model parameter sets to successfully construct a full or partial NN model.

[0201] (T) According to an embodiment, the syntax and semantics described below and shown in the following Table 34 are presented in the NNR layer parameter set. Table 34: NNR layer parameter set unit header syntax

[0202] The syntax element lps_independent_self_contained_flag equal to 0 indicates that the NNR unit referring to the layer parameter set is independently decodable and it should be possible to successfully reconstruct a full or partial NN model using the NNR unit. A value of 1 indicates that the NNR unit referring to the layer parameter set is not independently decodable and the payload of the NNR unit should be combined with NNR units referring to other layer parameter sets to successfully construct a full or partial NN model.

[0203] (U) According to an embodiment, the syntax and semantics described below and shown in the following Table 35 are presented in the NNR model parameter set. Table 35: NNR model parameter set unit header syntax

[0204] The syntax element mps_independently_decodable_flag equal to 0 indicates that the NNR units referencing the model parameter set are independently decodable. A value of 1 indicates that the NNR units referencing the model parameter set are not independently decodable.

[0205] The syntax element mps_self_contained_flag equal to 0 indicates that the NNR unit referencing the model parameter set is a model of a complete or partial NN that should be successfully reconstructed using the NNR unit. A value of 1 indicates that the NNR unit referencing the model parameter set should be combined with NNR units referencing other model parameter sets to successfully construct a complete or partial NN model.

[0206] (V) According to an embodiment, the syntax and semantics described below and shown in the following Table 36 are presented in the NNR layer parameter set. Table 36: NNR layer parameter set unit header

[0207] The syntax element lps_independently_decodable_flag equal to 0 indicates that the NNR units referring to the layer parameter set are independently decodable. A value of 1 indicates that the NNR units referring to the layer parameter set are not independently decodable.

[0208] The syntax element lps_self_contained_flag equal to 0 indicates that the NNR unit referencing the layer parameter set is a complete or partial NN model that should be successfully reconstructed using the NNR unit. A value of 1 indicates that the NNR unit referencing the layer parameter set should be combined with NNR units referencing other layer parameter sets to successfully construct a complete or partial NN model.

[0209] refer to Figure 6 , embodiments of the present disclosure may be implemented by a computing system 400. The computing system 400 may be configured to receive the NNR code stream 300 and may include at least one processor and a memory storing computer instructions. The computer instructions may be configured to cause the at least one processor to perform various aspects of the present disclosure when executed by the at least one processor.

[0210] For example, computing system 400 may implement a decoder, and computer instructions may include signaling / determination code 420 and decoding / reconstruction code 430 .

[0211] The signaling / determination code 420 may cause at least one processor to signal syntax elements and / or infer values ​​of syntax elements of various NNR units, and determine characteristics of the NNR units (and corresponding one or more neural networks) based on the syntax elements. The decoding / reconstruction code 430 may be configured to cause at least one processor to decode the encoded elements of the NNR codestream 300 and reconstruct one or more neural networks based on the decoding.

[0212] The above disclosure provides illustration and description, but is not intended to be exhaustive or limit the embodiments to the disclosed precise form. Modifications and variations may be made according to the above disclosure, or modifications and variations may be obtained from the practice of implementation. Embodiments of the present disclosure may include any number of the above-described aspects and / or additional aspects.

[0213] Obviously, the systems and / or methods described herein can be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited to these embodiments. Therefore, it should be understood that software and hardware can be designed to implement the systems and / or methods described herein.

[0214] Even though combinations of features are listed in the claims and / or described in the specification, these combinations are not intended to limit the disclosure of possible embodiments. In fact, many of these features can be combined in ways not specifically listed in the claims and / or described in the specification. Although each dependent claim listed below may be directly dependent on only one claim, the disclosure of possible embodiments includes each dependent claim in combination with every other claim in the claim set.

[0215] Elements, actions or instructions used herein should not be interpreted as critical or necessary unless explicitly described as such. Moreover, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.), and can be used interchangeably with "one or more". When only one item is meant, the term "one" or similar language is used. Moreover, as used herein, the terms "has", "have", "having" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".

Claims

1. A neural network encoding method, characterized in that: The method comprises: Obtaining a neural network with topology; Encoding the neural network having the topology to obtain a neural network representation NNR code stream; and Send the NNR code stream to the decoder; The NNR code stream includes a group of NNR units (GON) of the neural network, and the GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit and an NNR compression data unit, and Each of the NNR model parameter set unit, the NNR layer parameter set unit, the NNR topology unit, the NNR quantization unit and the NNR compression data unit is a corresponding at least one NNR unit of the GON, and each NNR unit includes a header and a payload.

2. The method according to claim 1, characterized in that The neural network representation NNR unit group GON is included in one or more aggregated NNR units of the NNR code stream, and the one or more aggregated NNR units each include an aggregated NNR unit header and an aggregated NNR unit payload, and the aggregated NNR unit payload includes at least a part of the NNR unit of the GON.

3. The method according to claim 2, characterized in that The neural network representation NNR unit group GON is comprised in a single aggregated NNR unit.

4. The method according to claim 3, characterized in that The single aggregated NNR unit includes a first syntax element indicating a type of the single aggregated NNR unit as a self-contained NNR aggregation unit.

5. The method according to claim 4, characterized in that The first syntax element is included in a header of the single aggregated NNR unit.

6. The method according to claim 1, characterized in that The NNR model parameter set unit includes a second syntax element indicating that an NNR unit referring to the NNR model parameter set unit is independently decodable.

7. The method according to claim 6, characterized in that The second syntax element is included in a header of the NNR model parameter set unit.

8. The method according to claim 1, characterized in that The NNR layer parameter set unit includes a third syntax element, and the third syntax element indicates that the NNR unit referring to the NNR layer parameter set unit is independently decodable.

9. The method according to claim 8, characterized in that The third syntax element is included in a header of the NNR layer parameter set unit.

10. The method according to claim 1, characterized in that In the neural network representation NNR unit group GON, the NNR model parameter set unit is followed by the NNR layer parameter set unit.

11. A neural network decoding method, characterized in that: include: receiving a neural network representation NNR code stream, wherein the NNR code stream comprises a group of NNR units (GON) of neural networks, and the GON comprises an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compression data unit, and Each of the NNR model parameter set unit, the NNR layer parameter set unit, the NNR topology unit, the NNR quantization unit, and the NNR compressed data unit is a corresponding at least one NNR unit of the GON, each NNR unit including a header and a payload; By decoding the NNR code stream, a neural network with topology is obtained.

12. A neural network coding system, characterized in that: include: at least one processor, and A memory storing computer code, wherein the computer code is configured to cause the at least one processor to execute the method according to any one of claims 1 to 10.

13. A non-volatile computer-readable medium storing computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the at least one processor performs the method according to any one of claims 1 to 11.

14. A computer device, characterized in that: The method comprises a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the method according to any one of claims 1 to 11.

15. A method for processing a video code stream, characterized in that: A video code stream is stored on a non-volatile computer-readable medium, wherein the video code stream is generated according to the encoding method according to any one of claims 1 to 10, or is decoded according to the decoding method according to claim 11.

Images