Methods, apparatus, devices, and computer-readable media for video encoding and decoding

Abstract

A method, apparatus, device, and non-transitory computer-readable medium are provided for video encoding. The non-transitory computer-readable medium stores computer code configured to cause at least one processor to perform the following operations: obtaining an input video stream; generating an encoded video bitstream based on the input video stream using a neural network, the encoded video bitstream including multiple auxiliary enhancement information (SEI) messages and blocks; determining multiple neural network topology information associated with the neural network; and signaling the determined multiple neural network topology information in multiple syntax elements associated with the encoded video bitstream.

Description

[0001] Cross-reference of related applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application No. 63 / 173,925, filed April 12, 2021, with the United States Patent and Trademark Office and U.S. Patent Application No. 17 / 499,007, filed October 12, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of video processing, and in particular to a method, apparatus, device for video encoding, and a non-transitory computer-readable medium. Background Technology

[0004] The ITU-T (International Telecommunication Union Telecommunication Standardization Sector), VCEG (Video Coding Expert Group, Q6 / 16), and ISO (International Organization for Standardization, ISO) / IEC (International Electrotechnical Commission, IEC) MPEG (Moving Picture Experts Group, JTC1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Revision 1), 2014 (Revision 2), 2015 (Revision 3), and 2016 (Revision 4). Since then, they have been investigating the potential need for standardization of future video coding technologies with compression capabilities significantly exceeding those of the HEVC standard (including its extensions). In October 2017, they released a joint proposal (Call for Proposal, CfP) on video compression capabilities exceeding those of HEVC. By February 15, 2018, a total of 22 CfP responses had been submitted regarding Standard Dynamic Range (SDR), 12 regarding High Dynamic Range (HDR), and 12 regarding 360 video categories. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET (Joint Video Exploration Team - Joint Video Expert Team) meeting. Through careful evaluation, JVET formally initiated the standardization of the next-generation video coding system beyond HEVC, namely Versatile Video Coding (VVC). Meanwhile, China's Audio Video Coding Standard (AVS) was also underway.

[0005] At least due to the complexity of neural network-based encoding, ordinary codecs may not perform filtering well when neural networks are involved. Therefore, there are technical drawbacks including a lack of compressibility, accuracy, and other unnecessary loss of information related to neural networks. Summary of the Invention

[0006] According to an exemplary embodiment, a method for video encoding is executed by at least one processor and includes: obtaining an input video stream; generating an encoded video bitstream based on the input video stream using a neural network, the encoded video bitstream including a plurality of auxiliary enhancement information (SEI) messages and blocks; determining a plurality of neural network topology information associated with the neural network; and signaling the determined plurality of neural network topology information in a plurality of syntax elements associated with the encoded video bitstream.

[0007] According to an exemplary embodiment, an apparatus for video encoding includes: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as instructed by the program code. The program code includes: acquisition code configured to cause the at least one processor to acquire an input video stream; encoding code configured to cause the at least one processor to generate an encoded video bitstream based on the input video stream using a neural network, the encoded video bitstream including a plurality of auxiliary enhancement information (SEI) messages and blocks; determination code configured to cause the at least one processor to determine a plurality of neural network topology information associated with the neural network; and signal notification code configured to cause the at least one processor to signal the determined plurality of neural network topology information in a plurality of syntax elements associated with the encoded video bitstream.

[0008] According to an exemplary embodiment, a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform the following operations: obtaining an input video stream; generating an encoded video bitstream based on the input video stream using a neural network, the encoded video bitstream including a plurality of auxiliary enhancement information (SEI) messages and blocks; determining a plurality of neural network topology information associated with the neural network; and signaling the determined plurality of neural network topology information in a plurality of syntax elements associated with the encoded video bitstream.

[0009] According to an exemplary embodiment, an apparatus for video encoding includes: an acquisition module configured to acquire an input video stream; an encoding module configured to generate an encoded video bitstream based on the input video stream using a neural network, the encoded video bitstream including a plurality of auxiliary enhancement information (SEI) messages and blocks; a determination module configured to determine a plurality of neural network topology information associated with the neural network; and a notification module configured to signal the determined plurality of neural network topology information in a plurality of syntax elements associated with the encoded video bitstream.

[0010] According to an exemplary embodiment, a method for video decoding is provided. The method is executed by at least one processor, and the method includes: receiving an encoded video bitstream generated using a neural network, the encoded video bitstream including a plurality of auxiliary enhancement information (SEI) messages and blocks; receiving a plurality of syntax elements associated with the encoded video bitstream, the plurality of syntax elements being used to signal determined plurality of neural network topology information; and determining the plurality of neural network topology information associated with the neural network.

[0011] According to the technical solution of this application, two mechanisms for notifying neural network models of information using signals are provided. In the first mechanism, one or more topological information and corresponding parameters trained using specific syntax elements defined in VSEI are explicitly notified using signals. In the second mechanism, external link information indicating the existence of corresponding topological information and network parameters is provided. Attached Figure Description

[0012] Figure 1 This is a simplified illustration based on a schematic diagram of an implementation method.

[0013] Figure 2 This is a simplified illustration based on a schematic diagram of an implementation method.

[0014] Figure 3 This is a simplified illustration based on a schematic diagram of an implementation method.

[0015] Figure 4 This is a simplified illustration based on a schematic diagram of an implementation method.

[0016] Figure 5 This is a simplified illustration based on the implementation method.

[0017] Figure 6 This is a simplified illustration based on the implementation method.

[0018] Figure 7 This is a simplified illustration based on the implementation method.

[0019] Figure 8 This is a simplified illustration based on the implementation method.

[0020] Figure 9A This is a simplified illustration based on the implementation method.

[0021] Figure 9B This is a simplified illustration based on the implementation method.

[0022] Figure 10 This is a diagram of the network structure according to the implementation method.

[0023] Figure 11This is an example illustration of a linear computation graph used in convolutional neural networks.

[0024] Figure 12 This is an example of block-level selection of NN (Neural Network) inference output.

[0025] Figure 13 This is a flowchart of model decision processing based on Rate-Distortion Optimization (RDO) in image / block-level models.

[0026] Figure 14 This is a diagram of an NN processing apparatus according to an embodiment.

[0027] Figure 15A This is a diagram showing a comparison of the inference processes of two neural networks.

[0028] Figure 15B yes Figure 15A The block diagram of the network model used for the two NN inference processes.

[0029] Figure 16 This is a simplified illustration of a flowchart based on an implementation method.

[0030] Figure 17 This is a simplified illustration of a flowchart based on an implementation method.

[0031] Figure 18 This is a simplified illustration of a flowchart based on an implementation method.

[0032] Figure 19 This is a simplified illustration of a flowchart based on an implementation method.

[0033] Figure 20 This is a simplified illustration of a flowchart based on an implementation method.

[0034] Figure 21 This is a simplified illustration of a flowchart based on an implementation method.

[0035] Figure 22 This is a simplified illustration of a flowchart based on an implementation method.

[0036] Figure 23 This is a simplified illustration based on a schematic diagram of an implementation method. Detailed Implementation

[0037] Implementations of this disclosure may relate to high-level syntax designs for video coding technologies such as HEVC (High Efficiency Video Coding), VVC (Video Coding), AV1&2, or the Audio Video Coding Standard (AVS). This disclosure proposes a high-level syntax design for carrying neural network topologies and parameters for post-filtering using a neural network model. This paper describes two SEI (Supplemental Enhancement Information) messages: 1) internal and external carriers describing the topology information and compressed network parameters; and 2) picture and / or block-level adaptive selection of the model. The proposed syntax structure is designed to be specified in the SEI as a codec-independent approach, but potentially similar syntax elements can be specified in parameter sets for VVC / HEVC / AV1&2 / AVS extensions, metadata tracks for file formats, or any other payload format.

[0038] The features discussed below can be used individually or in any combination in any order. Furthermore, implementations can be carried out by processing a circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0039] In VVC and AVS3, methods and devices based on neural networks have been proposed, particularly neural network-based filters. Below is one example of the structure of a neural network-based filter.

[0040] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present disclosure is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data at a local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0041] Figure 1 A second pair of terminals 101 and 104 is shown, which are provided to support bidirectional transmission of encoded video, which may occur, for example, during a video conference. For bidirectional data transmission, each terminal 101 and / or 104 can encode video data captured at a local location for transmission to the other terminal via network 105. Each terminal 101 and / or 104 can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0042] exist Figure 1 In this disclosure, terminals 101, 102, 103, and / or 104 may be shown as servers, personal computers, and smartphones, but the principles of this disclosure are not limited thereto. Implementations of this disclosure are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to a plurality of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 101, 102, 103, and 104. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of network 105 may be irrelevant to the operation of this disclosure.

[0043] Figure 2 The arrangement of a video encoder and video decoder in a streaming environment is illustrated as an example of an application to the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and storing compressed video on digital media including CDs (Compact Discs), DVDs (Digital Video Disks), Memory Sticks, etc.

[0044] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera device, that creates, for example, an uncompressed video sample stream 213. The sample stream 213 may be characterized by a high data volume when compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the camera device 201. The encoder 202 may include hardware, software, or a combination thereof to implement or enforce various aspects of the disclosed subject matter as described in more detail below. The encoded video bitstream 204 may be characterized by a lower data volume when compared to the sample stream and may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include video decoder 211, which decodes the incoming copy 208 of the encoded video bitstream and creates an outgoing video sample stream 210 that can be displayed on display 209 or another presentation device. In some streaming systems, the video bitstreams 204, 206, and / or 208 may be encoded according to certain video coding standards and / or video compression standards.

[0045] Figure 3 This can be a functional block diagram of the video decoder 300 according to the implementation method.

[0046] Receiver 302 can receive one or more codec video sequences to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Encoded video sequences can be received from channel 301, which can be a hardware / software link to a storage device storing the encoded video data. Receiver 302 can receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams, which can be forwarded to their respective user entities. Receiver 302 can separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 can be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter referred to as the "parser"). Buffer 303 may not be required or may be small when receiver 302 is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isoscillatory network. For use on best-effort packet networks such as the Internet, buffer 303 may be used, which can be relatively large and can advantageously have an adaptive size.

[0047] Video decoder 300 may include parser 304 to reconstruct symbols 313 from an entropy-encoded video sequence. These symbols may include information for managing the operation of decoder 300, and information potentially for controlling a presentation device such as display 312, which is not part of the decoder but may be coupled to it. Control information for the presentation device may be in the form of supplementary enhancement information (SEI messages) or fragments of video availability information parameter sets. Parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. Parser 304 may extract a subgroup parameter set from the encoded video sequence for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a group. Subgroups can include: Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0048] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbol 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler and / or the inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0049] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed by parser 304 from the encoded video sequence. For clarity, the flow of this subgroup control information between parser 304 and the following multiple units is not depicted.

[0050] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into multiple functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0051] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients and control information—including which transform to use, block size, quantization factor, quantization scaling matrix, etc.—from the parser 304 as symbol 313. The scaler / inverse transform unit 305 can output blocks containing sample values, which can be input to the aggregator 310.

[0052] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to an intra-coded block; that is, this block does not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding reconstructed information extracted from the current (partially reconstructed) image 309 to generate a block of the same size and shape as the reconstructed block. In some cases, the aggregator 310 adds the predictive information already generated by the intra-picture prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.

[0053] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coded blocks and potentially motion-compensated blocks. In this case, the motion-compensated prediction unit 306 can access the reference image memory 308 to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbol 313 belonging to the block, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) to generate output sample information. The address in the reference image memory from which the motion-compensated unit extracts its predicted samples can be controlled by motion vectors, which are provided to the motion-compensated unit in the form of symbols 313, which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory when the subsample precise motion vector is in use, motion vector prediction mechanisms, etc.

[0054] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video bitstream and available to loop filter unit 311 as symbols 313 from parser 304. However, video compression techniques may also respond to metadata obtained during decoding of previous (in decoding order) portions of encoded images or encoded video sequences, and to previously reconstructed and loop-filtered sample values.

[0055] The output of the loop filter unit 311 can be a sample stream, which can be output to the presentation device 312 and stored in the reference image memory 308 for use in future inter-frame image prediction.

[0056] Once fully reconstructed, certain encoded images can be used as reference images for future predictions. Once an encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and new current image memory can be reallocated before reconstructing subsequent encoded images begins.

[0057] The video decoder 300 can perform decoding operations according to predetermined video compression techniques that can be described in standards such as ITU-T Rec.H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in use, in the sense that the encoded video sequence follows the syntax specified in the video compression technique document or standard, and particularly in its configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata managed by the HRD buffer used for signaling in the encoded video sequence.

[0058] In this implementation, receiver 302 can receive supplemental (redundant) data and encoded video. The supplemental data may be included as part of the encoded video sequence. The supplemental data may be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0059] Figure 4 This may be a functional block diagram of a video encoder 400 according to an embodiment of this disclosure.

[0060] The encoder 400 can receive video samples from a video source 401 (which is not part of the encoder), which can capture video images to be encoded by the encoder 400.

[0061] Video source 401 can provide a source video sequence in the form of a digital video sample stream to be encoded by an encoder (400), the digital video sample stream having any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601YCrCb, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera device capturing local image information as a video sequence. Video data can be provided as multiple individual pictures that are given motion when viewed sequentially. The pictures themselves can be organized as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following focuses on describing samples.

[0062] According to the implementation, encoder 400 can encode and compress images of the source video sequence into an encoded video sequence 410 in real time or, as desired, under any other time constraints. Implementing an appropriate encoding rate is a function of controller 402. The controller controls and is functionally coupled to other functional units described below. For clarity, the coupling is not depicted. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of images (GOP) layout, maximum motion vector search range, etc. Other functions of controller 402 can be readily identified by those skilled in the art, as these functions may belong to the video encoder 400 optimized for a particular system design.

[0063] Some video encoders operate in a manner readily recognizable to those skilled in the art as an "encoding loop." As an oversimplification, an encoding loop may include: an encoding portion of encoder 400 (hereinafter referred to as the "source encoder") (responsible for creating symbols based on the input picture and reference picture to be encoded) and a (local) decoder 406 embedded in encoder 400, which reconstructs the symbols to create sample data that the (remote) decoder will also create (since any compression between the symbols and the encoded video bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory 405. Since the decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference picture buffer are also bit-precise between the local and remote encoders. In other words, the reference picture samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" during prediction. The basic principle of this reference picture synchronization (and the drift that occurs, for example, due to channel errors, when synchronization cannot be maintained) is well known to those skilled in the art.

[0064] The operation of the "local" decoder 406 can be combined with what has already been mentioned above. Figure 3 The operation is the same as that of the "remote" decoder 300 described in detail. However, a brief reference is also provided. Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 can losslessly encode / decode the symbols into an encoded video sequence, the entropy decoding portion of the decoder 300, which includes the channel 301, receiver 302, buffer 303, and parser 304, may not be fully implemented in the local decoder 406.

[0065] It can be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be omitted because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are provided below only in certain areas.

[0066] As part of its operation, the source encoder 403 can perform motion-compensated predictive coding, which references one or more previously encoded frames in the video sequence designated as "reference frames" to predictively encode the input frame. In this way, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as the prediction reference for the input frame.

[0067] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames based on symbols created by the source encoder 403. The operation of the encoding engine 407 can advantageously support lossy processing. When encoded video data can be decoded by the video decoder (… Figure 4 When decoded at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 can replicate the decoding process performed on the reference frame by the video decoder, and the reconstructed reference frame can be stored in the reference image memory 405. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame that will be obtained by the remote video decoder.

[0068] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata such as reference image motion vectors, block shapes, etc., that can be used as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on the sample blocks to find appropriate prediction references. In some cases, as determined by the search results obtained by predictor 404, the input image may have prediction references obtained from multiple reference images stored in reference image memory 405.

[0069] The controller 402 can manage the encoding operations of the video encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0070] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 408. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by lossless compression of the symbols using techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0071] Transmitter 409 can buffer the encoded video sequence created by entropy encoder 408 in preparation for transmission via communication channel 411, which can be a hardware / software link to a storage device where the encoded video data will be stored. Transmitter 409 can merge the encoded video data from video encoder 403 with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0072] The controller 402 can manage the operation of the encoder 400. During encoding, the controller 402 can assign a specific encoded image type to each encoded image, which may affect the encoding technique that can be applied to the corresponding image. For example, an image can typically be assigned to one of the following frame types:

[0073] An intra-frame picture (I-picture) can be a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, independent decoder refresh pictures. Those skilled in the art will understand these variations of I-pictures and their corresponding applications and characteristics.

[0074] Predictive images (P-images) can be images that are encoded and decoded using intra-frame or inter-frame prediction to predict sample values ​​for each block using at most one motion vector and a reference index.

[0075] Bidirectional predictive images (B-images) can be images encoded and decoded using intra-frame or inter-frame prediction to predict sample values ​​for each block using up to two motion vectors and reference indices. Similarly, multi-predictive images can use more than two reference images and associated metadata for the reconstruction of a single block.

[0076] Source images are typically spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignment of the corresponding images applied to the block. For example, blocks of an I-image can be nonpredictively coded, or blocks of an I-image can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be nonpredictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be nonpredictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0077] The video encoder 400 can perform encoding operations according to predetermined video coding techniques or video coding standards such as ITU-T Rec.H.265. During operation, the video encoder 400 can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or video coding standard used.

[0078] In this implementation, transmitter 409 may transmit additional data and encoded video. Source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, auxiliary enhancement information (SEI) messages, fragments of video usability information (VUI) parameter sets, etc.

[0079] Figure 5 The intra-prediction modes used in HEVC and JEM (Joint Exploration Model) are shown. To capture arbitrary edge orientations presented in natural video, the number of directional intra-prediction modes is expanded from 33 used in HEVC to 65. Additional directional modes in JEM on top of HEVC are depicted as follows: Figure 5 The dashed arrows in the image show that the planar and DC modes remain unchanged. These denser directional intra-prediction modes apply to all block sizes and both luma and chroma intra-prediction. Figure 5 As shown, the directional intra-prediction modes associated with odd-numbered intra-prediction mode indices, indicated by dashed arrows, are called odd-numbered intra-prediction modes. The directional intra-prediction modes associated with even-numbered intra-prediction mode indices, indicated by solid arrows, are called even-numbered intra-prediction modes. In this document, Figure 5 The directional intra-frame prediction mode indicated by the solid or dashed arrows in the image is also called the angle mode.

[0080] In JEM, a total of 67 intra-prediction modes are used for lumen intra-prediction. To encode using intra-modes, a Most Probable Mode (MPM) list of size 6 is constructed based on the intra-modes of neighboring blocks. If an intra-mode is not from the MPM list, a signaling flag is used to indicate whether the intra-mode belongs to the selected mode. In JEM-3.0, 16 selected modes exist, which are uniformly selected according to every four angular modes. In JVET-D0114 and JVET-G0060, 16 secondary MPMs are derived to replace the uniformly selected modes.

[0081] Figure 6 The N reference layers used for intra-frame directional mode are shown. There are block unit 611, segment A 601, segment B 602, segment C 603, segment D 604, segment E 605, segment F 606, first reference layer 610, second reference layer 609, third reference layer 608 and fourth reference layer 607.

[0082] In both HEVC and JEM, as well as some other standards such as H.264 / AVC (Advanced Video Coding, AVC), the reference samples used to predict the current block are limited to the nearest reference line (row or column). In multi-reference-line intra-prediction methods, for intra-directivity modes, the number of candidate reference lines (rows or columns) is increased from one (i.e., the nearest) to N, where N is an integer greater than or equal to one. Figure 7 A 4×4 prediction unit (PU) is used as an example to illustrate the concept of a multi-line intra-frame directivity prediction method. The intra-frame directivity pattern can arbitrarily select one of N reference layers to generate the predictor. In other words, the predictor p(x,y) is generated based on one of the reference samples S1, S2, ..., SN. A signal flag is used to indicate which reference layer is selected for the intra-frame directivity pattern. If N is set to 1, the intra-frame directivity prediction method is the same as the traditional method in JEM 2.0. Figure 6 In this document, reference lines 610, 609, 608, and 607 comprise six segments 601, 602, 603, 604, 605, and 606, as well as a top-left reference sample. In this document, the reference layer is also referred to as a reference line. The coordinates of the top-left pixel within the current block cell are (0,0), and the coordinates of the top-left pixel in the first reference line are (-1,-1).

[0083] In JEM, for the luma component, neighboring samples used for intra-prediction sample generation are filtered before generation processing. This filtering is controlled by a given intra-prediction mode and transform block size. If the intra-prediction mode is DC or the transform block size is equal to 4×4, neighboring samples are not filtered. If the distance between the given intra-prediction mode and the vertical (or horizontal) mode is greater than a predefined threshold, filtering is enabled. For neighboring sample filtering, a [1,2,1] filter and a bilinear filter are used.

[0084] The Position Dependent Intra Prediction Combination (PDPC) method is an intra prediction method that calls a combination of HEVC-style intra predictions using filtered boundary reference samples and unfiltered boundary reference samples. The following is the calculation of each predicted sample pred[x][y] located at (x,y):

[0085] pred[x][y]=(wL*R -1,y +wT*R x,-1 +wTL*R -1,-1 +(64-wL-wT-wTL)*pred[x][y]+32)>>6 (1)

[0086] Among them, Rx,-1 R -1,y Let R represent the unfiltered reference samples located above and to the left of the current sample (x, y), respectively, and R... -1,-1 This represents the unfiltered reference sample located at the top left corner of the current block. The weighting is calculated as follows:

[0087] wT=32>>((y<<1)>>shift) (2)

[0088] wL=32>>((x<<1)>>shift) (3)

[0089] wTL=-(wL>>4)-(wT>>4) (4)

[0090] Shift = (log2(width) + log2(height) + 2) >> 2 (5)

[0091] Figure 7 Figure 700 illustrates the DC mode PDPC weighting (wL, wT, wTL) for the (0,0) and (1,0) positions within a 4×4 block. If the PDPC is applied to DC intra-frame mode, planar intra-frame mode, horizontal intra-frame mode, and vertical intra-frame mode, no additional boundary filters, such as the HEVC DC mode boundary filter or the horizontal / vertical mode edge filter, are required. Figure 7 The definitions of reference samples Rx,-1, R-1,y, and R-1,-1 for the PDPC applied to the top-right diagonal mode are shown. The predicted sample pred(x',y') is located at (x',y') within the predicted block. The coordinates x of the reference sample Rx,-1 are given by x = x' + y' + 1, and the coordinates y of the reference sample R-1,y are similarly given by y = x' + y' + 1.

[0092] Figure 8 Local Illumination Compensation (LIC) is shown in Figure 800, and is based on a linear model for illumination variations using scaling factor a and offset b. LIC is adaptively enabled or disabled for each coding unit (CU) coded for each inter-frame mode.

[0093] When applying LIC to a CU, the least squares error method is used to derive parameters a and b by using the neighboring samples of the current CU and their corresponding reference samples. More specifically, as... Figure 8 As shown, neighboring samples of the CU after secondary sampling (2:1 secondary sampling) and corresponding samples in the reference image (identified by the motion information of the current CU or sub-CU) are used. IC parameters are derived and applied to each prediction direction.

[0094] When encoding a CU using the merge mode, the LIC flag is copied from the adjacent block in a manner similar to motion information copying in the merge mode; otherwise, the CU is signaled with the LIC flag to indicate whether the LIC is applied.

[0095] Figure 9A The intra-prediction mode 900 used in HEVC is shown. In HEVC, there are a total of 35 intra-prediction modes, of which mode 10 is a horizontal mode, mode 26 is a vertical mode, and modes 2, 18, and 34 are diagonal modes. The intra-prediction mode is signaled using three most probable modes (MPMs) and 32 remaining modes.

[0096] Figure 9B The implementation of VVC shows a total of 87 intra-prediction modes, where mode 18 is a horizontal mode, mode 50 is a vertical mode, and modes 2, 34, and 66 are diagonal modes. Modes-1 through-10 and modes 67 through-76 are referred to as Wide-Angle Intra Prediction (WAIP) modes.

[0097] Based on the PDPC expression, a linear combination of the intra-frame prediction mode (DC, plane, angle) and the reference sample is used to predict the predicted sample pred(x,y) located at position (x,y):

[0098] pred(x,y)=(wL×R-1,y+wT×Rx,-1–wTL×R-1,-1+(64–wL–wT+wTL)×pred(x,y)+32)>>6 (6)

[0099] Where Rx,-1 and R-1,y represent reference samples located above and to the left of the current sample (x,y), respectively, and R-1,-1 represents a reference sample located at the top left corner of the current block.

[0100] For DC mode, weights are calculated as follows for blocks with width and height dimensions:

[0101] wT=32>>((y<<1)>>nScale),wL=32>>((x<<1)>>nScale),wTL=(wL>>4)+(wT>>4) (7) where, nScale=(log2(width)–2+log2(height)–2+2)>>2, where, wT represents the weighting factor of the reference sample with the same horizontal coordinate located within the upper reference line, wL represents the weighting factor of the reference sample with the same vertical coordinate located within the left reference line, and wTL represents the weighting factor of the upper left reference sample of the current block, nScale specifies the rate at which the weighting factor decreases along the axis (wL decreases from left to right or wT decreases from top to bottom), that is, the weighting factor deceleration rate, and in the current design, the weighting factor deceleration rate is the same along the x-axis (from left to right) and the y-axis (from top to bottom). Furthermore, 32 represents the initial weighting factor of the adjacent samples, and the initial weighting factor is also the top (left or top left) weighting assigned to the top left sample in the current CB, and the weighting factor of the adjacent samples in the PDPC process should be equal to or less than the initial weighting factor.

[0102] For planar mode, wTL = 0; for horizontal mode, wTL = wT; and for vertical mode, wTL = wL. PDPC weights can be calculated using only addition and shifting. The value of pred(x,y) can be calculated in a single step using Equation 1.

[0103] The methods presented herein can be used individually or in any combination in any order. Furthermore, each of the methods (or implementations), encoders, and decoders can be implemented using a processing circuitry system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. According to the implementation, the term "block" can be interpreted as a prediction block, coding block, or coding unit, i.e., a CU.

[0104] Figure 10 This is a diagram of the network structure 1000 according to the implementation method.

[0105] In the context of neural network-based coding, network architecture 1000 in VVC and AVS3 involves various neural network-based methods, particularly neural network-based filters. Network architecture 1000 represents a neural network-based filter that includes several convolutional layers. For example, the kernel size is 3*3*M, meaning that for each channel, the convolutional kernel size can be 3*3, and the number of output layers is M.

[0106] In network architecture 1000, combining convolutional layers and nonlinear activation functions (e.g., ReLU) allows the entire process to be viewed as a nonlinear filter for reconstruction, and the quality can be improved after filtering.

[0107] According to the implementation, the network structure 1000 can be a simplified block diagram, and considering the complexity of neural network-based encoding methods, ordinary codecs may not be able to perform filtering processing. Therefore, according to the exemplary implementation herein, several identifiers can be added to the SEI to indicate whether the current CVS (Concurrent Versions System, CVS) uses neural network-based tools. Furthermore, network details can also be indicated. Therefore, if the decoder cannot process neural network-based filters, information related to the neural network may be discarded; otherwise, the processing may be skipped.

[0108] As discussed further below, exemplary implementations provide at least two mechanisms for signaling information about the neural network model. The first mechanism is to explicitly signal one or more topological information elements and corresponding parameters trained using specific syntax elements defined in the VSEI (Versatile Supplemental Enhancement Information). The second mechanism is to provide external link information indicating the location where the corresponding topological information and network parameters exist.

[0109] For signaling network topology and parameters, existing formats already developed for representing networks can be used, according to exemplary implementations. Examples could involve the Neural Network Exchange Format (NNEF), a generalized neural network exchange format developed by Khronos. Other possible examples include implementations such as those involving Open Neural Network Exchange (ONNX) and MPEG NNR (Neural Network Compression standard, NNR), which are formats for coded representations of neural networks.

[0110] Ideally, any neural network model can be exported to NNEF and other formats, and network accelerators and libraries can use data in these formats without compatibility issues with any network framework. As a practical approach, implementations can directly reference external files or bitstreams with URI (Uniform Resource Identifier) ​​information. However, a lightweight syntax design is also desired to represent video coding-specific networks for VVC or HEVC extensions using new neural network-based video coding tools, as general representations of network models can be too large to be used for compressing video formats. Since most network models used for video compression are based on Convolutional Neural Networks (CNNs), a compact representation of CNNs in the SEI message is expected to help reduce the overall bitrate and facilitate easy access to network model data, according to the exemplary implementation.

[0111] The implementation described in this paper can be represented by a computational graph, which is a directed graph with multiple nodes. Nodes include operational nodes and data, such as tensors. Various network topologies are designed and used depending on the application. However, for post-loop / in-loop filtering in video processing, a simple CNN-based model is commonly used. In this case, a simple multi-layer feedforward network like a CNN can be represented by a linear graph starting from the input data, where each operational node in the layer produces intermediate processing data. Finally, output data is generated through multiple layers.

[0112] Figure 11 This is an example illustration of linear computation for CNNs.

[0113] like Figure 11 As shown, when input tensor data is fed to the operation node, the operation node processes the input tensor data using pre-trained constants and / or variables, and outputs intermediate tensor data (in... Figure 11 (Represented as a tensor in the original text). When an operation node is executed, the actual data is used by that operation node. Typically, for a CNN, the weighted sum of the input data and trained constants and / or updated variables is the output of each operation node.

[0114] Once a specific operation node is designated as a single step, the same operation node can be used iteratively. This simple network topology can be described using a few well-designed syntactic elements in the SEI message. For more complex model designs, external formats like NNEF or ONNX can be used.

[0115] To transmit network parameters, the amount of data from trained network parameters can often be too large to be contained in an SEI message. To reduce the data size, the MPEG-NNR format can be used to compress the parameters, which can then be divided into multiple data blocks. Each block of compressed parameters can be included in the SEI message or in a separate data file, which may be transmitted in the same bitstream or stored separately on a remote server. During decoding, according to an exemplary implementation, all cascaded data blocks in the SEI message used to represent the neural network are concatenated and used by a neural network library or decoder.

[0116] In example implementations, the bitstream may need to carry multiple NN topologies and their parameters, where the bitstream has multiple applications utilizing NN information, or applications utilizing multiple NNs. For example, JVET-U0087 uses three NN models for an application. To accommodate three NN models, the bitstream is allowed to contain multiple SEI messages carrying NN topologies and their parameters. One set of NN topologies and parameters can be included in the band, while other NN topologies and parameters are transmitted out of band. NN identifiers can be used to indicate the associated NN for each application. When NN inference processing utilizes multiple NNs, each picture or block may use a different NN topology and its parameters. In this case, each picture or block has its own NN identifier and on / off flag.

[0117] Figure 12 This is an example of block-level selection of NN inference output. Figure 12 Different NN inference processes applied to each block are illustrated. According to the implementation, the input image 1200 for NN processing can be processed using different NN inference processes. For example, in... Figure 12 The diagram shows output images 1201, 1202, and 1203 from three different NN inference processes. Implementations may include more or fewer NN inference processes. To address the different NN inference processes applied to each block, an implementation proposes using another SEI message to carry image / block-level adaptation information for multiple NNs, resulting in output image 1204 having block-level selections of output images (1201, 1202, 1203) from different inference processes. An example model of the block-level selection decision-making process according to the implementation will now be described.

[0118] Figure 13 This is a flowchart of the model decision processing based on Rate Distortion Optimization (RDO) in image / block-level models.

[0119] Figure 13The model decision processing includes image / block-level model decision module 1300, block size decision 1310, image-level model decision 1320, block-level model decision 1330, image / block size decision 1340, block-level on / off decision 1350, and image-level on / off decision 1360.

[0120] All RDO decision processing is performed through recursive calls from higher to lower levels. For example, at the highest level (i.e., picture / block-level model decision module 1300), the optimal mode is selected between picture-level model decision 1320 and block-level model decision 1330. To determine which mode is optimal between block-level model decision 1330 and picture-level model decision 1320, lower-level processing is called. For block-level model decision 1330, block size decision 1310 compares various block sizes. To select the optimal block size, multiple block-level models are compared to each other based on rate-distortion (RD) cost. Once the optimal block-level model is selected, the minimum cost of the block-level model for each block size can be calculated. Then, by comparing the minimum costs of various block sizes, the minimum cost of the block-level model pattern can be calculated. Similarly, the minimum cost of image-level model patterns can be computed through recursive and hierarchical call processing: image / block segmentation into various sizes of image / block size decisions 1340, block-level on / off decisions 1350, and image-level on / off decisions 1360.

[0121] Implementations of this disclosure propose that the reconstructed image used for output and display may differ from the reconstructed image used for motion compensation prediction. A first set of filters (including neural network-based filters) may be applied to the reconstructed image to create a version for output. When performing motion compensation, a reference image is created by applying a second set of filters (including neural network-based filters) to an image stored in a Decoded Picture Buffer (DPB).

[0122] Methods and apparatus for using neural networks for time filtering will now be described.

[0123] Figure 14 This is a diagram of an NN processing apparatus according to an embodiment.

[0124] like Figure 14 As shown, the device includes a decoding CTU (Coding Tree Unit) module 1400, a deblocking filter module 1410, a first filter module 1420 based on a neural network (NN), a decoding image buffer (DPB) 1430, a second filter module 1440 based on a neural network (NN), and a motion compensation module 1450.

[0125] The NN-based temporal processing of this implementation comprises two stages: NN-based reconstruction processing and NN-based prediction processing. Given a codec bitstream as input, the codec bitstream is decoded in the decoding CTU module 1400. Immediately after deblocking processing in the deblocking filter module 1410, the NN-based first reconstruction filter is processed by the NN-based first filtering module 1420 to improve the details of the decoded image to obtain an output image with better visual quality. Forward and backward reference images are used for network inference processing. After the decoded image is stored in the decoded image buffer (DPB) 1430, the NN-based second filtering module 1440 modifies the decoded image with reference pixel values ​​to better predict subsequent images in the decoding order. When motion compensation is performed, motion compensation prediction processing is performed in the motion compensation module 1450. The output of the motion compensation module 1450 is combined with the decoding output of the decoding CTU module 1400 and sent to the deblocking filter module 1410. This process can be further described here.

[0126] Figure 15A A comparison of two NN inference processes is shown: NN-based reconstruction processing 1510 and NN-based prediction processing 1520. Both the NN-based reconstruction and prediction processes use the same network model 1500, but with different training parameters.

[0127] Figure 15B This is a block diagram of network model 1500. The network model includes a PCD (Pyramid Cascading and Deformable) alignment module 1530 for spatial and temporal optimization, and a TSA for applying attention to emphasize important features for subsequent recovery.

[0128] (Temporal and Spatial Attention) Fusion Module 1540.

[0129] The NN-based reconstruction process 1510 is an NN-based reconstruction filter that has been trained to minimize the error between the reconstructed pixel values ​​of the current image and the original pixel values. An output image is generated based on the current decoded image and two reference images (one forward reference and one backward reference) stored in the decoded image buffer (DPB) 1430. The output image is an enhanced image with fused temporal features. The output image (reconstructed image) is stored in the decoded image buffer and used as one of the reference images to generate the next enhanced output image. The NN-based prediction process 1520 is a prediction filter that has been trained to minimize the error between the pixel values ​​of subsequent images in the decoding order and the reference pixel values. The reference image is updated by filtering it using the NN-based prediction process. The updated reference image is stored in the decoded image buffer.

[0130] An example of an SEI message for carrying NN information, according to an implementation, will now be described. Although the example assumes that syntax elements and parameters are signaled in one or more SEI messages, any parameter set (e.g., SPS (Sequence Parameter Set), PPS (Picture Parameter Set), APS (Adaptive Parameter Set),) any metadata track of a file format, or any payload type can carry the same or slightly modified syntax elements and parameters.

[0131] An example of neural network (NN) topology and parameter SEI message syntax according to the implementation will now be described. Figure 16 An exemplary flowchart 1600 illustrates aspects of the NN topology and parameter SEI messages and their syntax.

[0132] According to the implementation method, such syntax can be represented in Table 1 below:

[0133] Table 1

[0134]

[0135]

[0136]

[0137] nn_id provides an identifier for the neural network.

[0138] An nn_partition_flag value of 0 indicates that all data and trained parameters of the network topology are included in the SEI message, while an nn_partition_flag value of 1 indicates that the data and trained parameters of the network topology are split into multiple SEI messages.

[0139] An nn_output_pic_format_present_flag value of 0 indicates that the syntax elements of the output image format do not exist in the SEI message, and that the output image format of the neural network inference process is the same as the output image format of the decoder. An nn_output_pic_format_present_flag value of 1 indicates that the syntax elements of the output image format exist in the SEI message.

[0140] nn_postfilter_type_idc specifies the type of postfilter for the neural network represented by the SEI message, as specified in Table 2 (NN Postfilter Types) below.

[0141] Table 2

[0142] 0 Visual quality improvement in the case of a single input image 1 Visual quality improvement in the case of multiple input images 2 Super-resolution in the case of a single input image 3 Super-resolution in the case of multiple input images 4..15 reserve

[0143] `num_nn_input_ref_pic` specifies the number of input reference images. `num_nn_input_ref_pic` equal to 0 indicates that the current output image of the decoder is the only input data for the neural network, while `num_nn_input_ref_pic` greater than 0 indicates that the number of reference images used as input data for the neural network is `num_nn_input_ref_pic-1`.

[0144] num_partitioned_nn_sei_messages specifies the number of post-filtered SEI messages based on the neural network to represent the entire neural network topology and corresponding parameters, and when it does not exist, the value of num_partitioned_nn_sei_messages is inferred to be equal to 1.

[0145] nn_sei_message_idx specifies the index of the partial neural network data carried in the SEI message, and when it does not exist, the value of nn_sei_message_idx is inferred to be equal to 0.

[0146] Given the above syntax, flowchart 1600 shows that at S1601, the post-filter can be initialized and data S1602 can be generated or obtained, such that at S1603, it can then be determined whether an information flag such as network_topology_info_external_present_flag exists, such that if the information flag exists, data including external_nn_topology_info_format_idc at S1604, num_bytes_external_network_topology_uri_info at S1605, and external_nn_topology_uri_info at S1606 can be obtained; otherwise, a check can be performed at S1607 to receive network_topology_info(input).

[0147] In other words, according to the exemplary implementation, nn_topology_info_external_present_flag equal to 0 indicates that the data of the neural network topology representation is included in the SEI message, while nn_topology_info_external_present_flag equal to 1 indicates that the data of the neural network topology representation may exist externally, and the SEI message only contains external link information.

[0148] External_nn_topology_info_format_idc, such as at S1604, can specify the external storage format for the neural network topology representation, as specified in Table 3 below (External NN Topology Information Format Identifiers):

[0149] Table 3

[0150] 0 Unrecognized storage format 1 NNEF 2 ONNX 3..15 reserve

[0151] For example, num_bytes_external_network_topology_uri_info in S1605 specifies the number of bytes in the syntax element external_network_topology_uri_info.

[0152] For example, external_nn_topology_uri_info in S1606 specifies the URI information for external neural network topology. The length of the syntax element can be Ceil(Log2(num_bytes_external_nn_topology_uri_info)) bytes.

[0153] For example, network_topology_info(input) at S1607 may involve processing according to Table 4 below.

[0154] Table 4

[0155]

[0156]

[0157] For example, see Figure 17 The exemplary flowchart 1700, in which at S1701 it is determined that there is a process based on network_topology_info(input), which may continue to generate or obtain one or more of the following: nn_topology_storage_format_idc at S1702, nn_topology_compression_format_idc at S1703, num_bytes_topology_data at S1704, and at S1705 it is determined whether nn_top_format_idc>0 exists. If it is determined at S1705 that nn_top_format_idc>0, then at S1706, nn_topology_data_byte[I] is obtained.

[0158] For example, nn_topology_storage_format_idc at S1702 specifies the storage format of the neural network topology representation, as specified in Table 5 (NN topology storage format identifiers) below:

[0159] Table 5

[0160] 0 Unrecognized storage format 1 NNEF 2 ONNX 3..15 reserve

[0161] The nn_topology_compression_format_idc at S1703 specifies the compression format of the neural network topology, as specified in Table 6 (NN Topology Compression Format Identifiers) below.

[0162] Table 6

[0163] 0 Uncompressed 1 ZLIB compressed data format 2..15 reserve

[0164] For example, num_bytes_topology_data at S1704 specifies the number of bytes of neural network topology payload contained in this SEI message.

[0165] For example, nn_topology_data_byte[I] at S1706 specifies the i-th byte of the neural network topology payload.

[0166] For example, num_variables at S1708 specifies the number of variables that can be used to perform operations on nodes in the neural network specified by the SEI message.

[0167] For example, num_node_types at S1709 specifies the number of operator node types that can be used to execute the operator nodes in the neural network specified by the SEI message.

[0168] For example, num_operation_node_executions at S1710 specifies the number of times the operation node is executed using the input variables of the neural network specified by this SEI message.

[0169] Return to Figure 16 In flowchart 1600, at S1608, it can be determined whether network_parameter_info_external_present_flag exists. If it exists, external_network_parameter_info_format_idc at S1609, num_bytes_external_network_parameter_uri_info at S1610, and external_nn_parameter_uri_info at S1611 can be generated or obtained. Otherwise, at S1612, network_parameter_info(input) can be obtained or generated.

[0170] For example, in S1608, network_parameter_info_external_present_flag equal to 0 indicates that the data of the neural network parameters is constrained in the SEI message, while network_parameter_info_external_present_flag equal to 1 indicates that the data of the neural network parameters may exist externally, and the SEI message only contains external link information.

[0171] The external_network_parameter_info_format_idc field at S1609 specifies the external storage format for neural network parameters, as specified in Table 7 below (External NN Parameter Storage Format Identifiers):

[0172] Table 7

[0173] 0 Unrecognized storage format 1 NNEF 2 ONNX 3 MPEG-NNR 4..15 reserve

[0174] The num_bytes_external_network_parameter_uri_info field in S1610 specifies the number of bytes in the syntax element external_network_parameter_uri_info.

[0175] The external_nn_parameter_uri_info at S1611 specifies the URI information for the external neural network parameters. The length of the syntax element is Ceil(Log2(num_bytes_external_network_parameteruri_info)) bytes.

[0176] For example, network_parameter_info(input) at S1612 indicates the following processing in Table 8:

[0177] Table 8

[0178]

[0179] As in Figure 18 In flowchart 1800, S1801—which also represents Figure 16 The determination of network_parameter_info(input) at S1612 involves obtaining or otherwise generating nn_parameter_type_idc at S1802, nn_parameter_storage_format_idc at S1803, nn_parameter_compression_format_idc at S1804, num_bytes_parameter_data at S1805, and nn_parameter_data_byte at S1806.

[0180] For example, nn_parameter_type_idc at S1802 specifies the data payload type of the neural network parameters, as specified in Table 9 below (NN parameter payload types):

[0181] Table 9

[0182] 0 Integer 1 buoy 2..15 reserve

[0183] The nn_parameter_storage_format_idc parameter at S1803 specifies the storage format of the neural network parameters, as specified in Table 10 (NN parameter storage format identifiers) below:

[0184] Table 10

[0185] 0 Unrecognized storage format 1 NNEF 2 ONNX 3 MPEG-NNR 4..15 reserve

[0186] For example, nn_parameter_compression_format_idc at S1804 specifies the compression format of the neural network parameters, as specified in Table 11 (NN topology compression format identifiers) below:

[0187] Table 11

[0188] 0 Uncompressed 1 ZLIB compressed data format 2..15 reserve

[0189] For example, num_bytes_parameter_data at S1805 specifies the number of bytes of neural network parameter payload contained in this SEI message.

[0190] For example, nn_parameter_data_byte at S1806 specifies the i-th byte of the neural network parameter payload.

[0191] Return to Figure 16 At S1613, the process can continue to, for example, Figure 19 In S1901 of flowchart 1900, it can be determined whether network_input_pic_format_present_flag exists. If it exists, nn_input_chroma_format_idc at S1902, nn_input_bitdepth_minus8 at S1903, nn_input_pic_width at S1904, nn_input_pic_height at S1905, and at S1906, it can be determined whether nn_patch_size_present_flag exists. If it exists, nn_input_patch_width at S1907, nn_input_patch_height at S1908, and nn_boundary_padding_idc at S1909 can be obtained or generated.

[0192] For example, a value of 0 for `network_input_pic_format_present_flag` in S1901 indicates that the syntax elements for the input image format do not exist in the SEI message, and that the input image format for neural network inference is the same as the output image format for the decoder. A value of 1 for `nn_input_pic_format_present_flag` indicates that the syntax elements for the input image format exist in the SEI message.

[0193] At S1902, nn_input_chroma_format_idc can specify the chroma sampling relative to the luminance sampling according to Table 12 (chroma format identifiers):

[0194] Table 12

[0195] 0 monochrome 1 4:2:0 2 4:2:2 3 4:4:4

[0196] nn_input_bitdepth_minus8 (or nn_input_bitdepth_plus 8) at S1903 specifies the bit depth of the luminance and chrominance samples in the input image of the neural network.

[0197] For example, nn_input_pic_width in S1904 specifies the width of the input image.

[0198] For example, nn_input_pic_height in S1905 specifies the height of the input image.

[0199] For example, in S1906, nn_patch_size_present_flag equal to 0 specifies that the patch size is equal to the input image size. nn_patch_size_present_flag equal to 1 specifies that the patch size is explicitly signaled.

[0200] For example, nn_input_patch_width in S1907 specifies the width of the patch used for neural network inference processing.

[0201] For example, nn_input_patch_height at S1908 specifies the height of the patch used for neural network inference processing.

[0202] For example, nn_boundary_padding_idc at S1909 specifies the padding method applied to the boundary of the tile when the tile size is different from the input image size, for example, according to Table 13 (boundary padding identifiers) below:

[0203] Table 13

[0204] 0 Nearest neighbor copy 1..3 reserve

[0205] Return to Figure 16 At S1613, this process can continue in parallel or serially until... Figure 19 Flowchart 1900 and Figure 20 The flowchart 2000 shows that at S2001, it can be determined whether num_network_input_ref_pic>0 exists. If it exists, then num_fwd_ref_pics_as_input at S2002, an indication that NumFwdRefPics>0 exists at S2003, and an indication that NumBwdRefPics>0 exists at S2006 can be obtained or generated. Furthermore, if an indication that NumFwdRefPics>0 exists is determined at S2003, then nearest_fwd_ref_pics_as_input at S2004 and poc_dist_fwd_ref_pic[i] at S2005 can be determined. Furthermore, if an indication that NumBwdRefPics>0 is found at S2006, then the nearest_bwd_ref_pics_as_input at S2007 and the poc_dist_bwd_ref_pic[i] at S2008 can be determined.

[0206] For example, num_fwd_ref_pics_as_input at S2002 specifies the number of forward reference pictures used as input data for the neural network, for example, if (num_nn_input_ref_pic>0) then (NumFwdRefPics=num_fwd_ref_pics_as_input) otherwise (NumFwdRefPics=0).

[0207] For example, in S2004, the nearest_fwd_ref_pics_as_input specifies the nearest forward reference image that has the smallest image sequence count distance from the current image as the input data for the neural network.

[0208] For example, in S2005, poc_dist_fwd_ref_pic[i] specifies the image order count of the i-th forward reference image used as input data for the neural network. The image order count of the i-th forward reference image is equal to the image order count of the current image minus poc_dist_fwd_ref_pic[i].

[0209] For example, nearest_bwd_ref_pics_as_input in S2007 specifies the number of backward reference pictures used as input data for the neural network, such that if (num_nn_input_ref_pic>0) then (NumBwdRefPics=num_bwd_ref_pics_as_input) otherwise (NumBwdRefPics=0).

[0210] For example, poc_dist_bwd_ref_pic[i] in S2008 specifies the image order count of the i-th backward reference image used as input data for the neural network. The image order count of the i-th backward reference image is equal to the image order count of the current image plus poc_dist_bwd_ref_pic[i].

[0211] In addition, as a note, the nearest_bwd_ref_pics_used_flag in Table 1 above specifies the nearest backward reference image that has the smallest image order count distance from the current image as the input data for the neural network.

[0212] For example, it can be used Figure 21 The additional operations are implemented in flowchart 2100. In flowchart 2100, there is define_operation_node(i) at S2101, which iteratively defines the following operations: nn_operation_class_idc[i] at S2102, nn_operation_function_idc[i] at S2103, num_input_variables[i] at S2104, and num_output_variables[i] at S2105.

[0213] For example, nn_operation_class_idc[i] at S2102 specifies the class of the i-th operation node, as specified in Table 14 (NN operation functions) below:

[0214] Table 14

[0215] 1 Convolution / Deconvolution 2 Filtering 3 Upsampling 4 collection 5 element-wise operations 6 Reshape 7 activation 8 Quantification 9..255 reserve

[0216] For example, nn_operation_function_idc[i] at S2103 specifies the function of the i-th operation node, as specified in Table 15 below (NN operation functions, example Table 15 (activation functions) where nn_operation_class_idc equals 7):

[0217] Table 15

[0218]

[0219] For example, num_input_variables[i] in S2104 specifies the number of input variables for the i-th operation node.

[0220] For example, num_output_variables[i] at S2105 specifies the number of output variables for the i-th operation node.

[0221] The syntax of Figure 15 can be represented by the following Table 16:

[0222] Table 16

[0223]

[0224] For example, it can be used Figure 22 The additional operations are implemented in flowchart 2200. In flowchart 2200, there is operation_node_execution(i) at S2201, which iteratively defines the following operations: nn_op_node_idx[i] at S2202, nn_input_variable_idx[i][j] at S2203, and nn_output_variable_idx[i][j] at S2204.

[0225] For example, nn_op_node_idx[i] at S2202 specifies the index of the operation node to be executed by the i-th operation node. For this execution, the nn_op_node_idx[i]-th operation node is used.

[0226] For example, nn_input_variable_idx[i][j] at S2203 specifies the variable index of the j-th input variable executed by the i-th operation node.

[0227] For example, nn_output_variable_idx[i][j] at S2204 specifies the variable index of the j-th output variable executed by the i-th operation node.

[0228] about Figure 22 The syntax can be represented by the following Table 17:

[0229] Table 17

[0230]

[0231] Additional processing may involve, for example, utilizing Figure 21 For example, define the variable (i) iteratively according to the following syntax in Table 18:

[0232] Table 18

[0233]

[0234] Referring to Table 18, nn_variable_class_idc[i] specifies the variable class of the i-th variable in the neural network, as specified in Table 19 (NN Variable Classes) below:

[0235] Table 19

[0236] 0 not specified 1 enter 2 Output 3 intermediate variables 4 nuclear 5 bias 6 constant 7..15 reserve

[0237] According to the exemplary implementation, it can be determined that when nn_variable_class_idc equals 1, the variable is the input data of the neural network; when nn_variable_class_idc equals 2, the variable is the output data of the neural network; when nn_variable_class_idc equals 3, the variable is the intermediate data between operation nodes; and when nn_variable_class_idc equals 4, the variable is pre-trained or predefined constant data.

[0238] Furthermore, referring to Table 18, nn_variable_type_idc[I] specifies the variable type of the i-th variable in the neural network, as specified in Table 20 (NN Variable Types) below:

[0239] Table 20

[0240] 0 Integer 1 buoy 2..15 reserve

[0241] According to an exemplary implementation, nn_variable_dimensions[I] in Table 18 specifies the number of dimensions of the i-th variable, and furthermore, nn_variable_dimension_size[I][j] specifies the size of the j-th dimension of the i-th variable. As a note, according to an exemplary implementation, when the i-th variable is input data with 3, 1920, and 1080 color components, the values ​​are: nn_variable_class_idc[i] equals 1, nn_variable_dimensions[I] equals 3, nn_variable_dimension_size[I][0] equals 3, nn_variable_dimension_size[I][1] equals 1920, and nn_variable_dimension_size[I][2] equals 1080.

[0242] An example of the neural network inference processing SEI message syntax according to the implementation method will now be described.

[0243] According to the implementation method, such syntax can be represented in Table 21 below:

[0244] Table 21

[0245]

[0246]

[0247] The increment of 1 in num_nn_models_minus1 specifies the number of neural network models used for the image.

[0248] nn_used_id[i] indicates the identifier of the i-th neural network model used for the image.

[0249] nn_pic_width_in_luma_samples specifies the width of each decoded image associated with the SEI message, in units of luminance samples, and nn_pic_height_in_luma_samples specifies the height of each decoded image associated with the SEI message, in units of luminance samples.

[0250] `nn_num_block_rows_minus1` plus 1 specifies the number of block row heights. In implementations, the value of `nn_num_block_rows_minus1` can be in the range of 0 to `nn_pic_height_in_luma_samples-1` (inclusive of endpoints). `nn_num_block_columns_minus1` plus 1 specifies the number of block column widths. In implementations, the value of `nn_num_block_columns_minus1` can be in the range of 0 to `nn_pic_width_in_luma_samples-1` (inclusive of endpoints).

[0251] `nn_block_column_width_minus1[i]` incremented by 1 specifies the width of the i-th block in units of luminance samples. In an implementation, the value of `nn_block_column_width_minus1[i]` can be in the range of 0 to `nn_pic_width_in_luma_samples-1` (inclusive of endpoints). `nn_block_row_height_minus1[i]` incremented by 1 specifies the height of the i-th block in units of luminance samples. In an implementation, the value of `nn_block_row_height_minus1[i]` can be in the range of 0 to `nn_pic_height_in_luma_samples-1` (inclusive of endpoints).

[0252] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media, or can be implemented through one or more specially configured hardware processors. For example, Figure 23 A computer system 2300 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0253] Computer software can be encoded using any suitable machine code or computer language, which can be subjected to mechanisms such as assembly, compilation, and linking to create code including instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, etc.

[0254] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0255] Figure 23 The components shown for computer system 2300 are exemplary in nature and are not intended to limit the scope or functionality of the computer software used in embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any component or combination of components shown in the exemplary embodiments of computer system 2300.

[0256] Computer system 2300 may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement with a data glove), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input. Human-machine interface devices can also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0257] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard 2301, mouse 2302, touchpad 2303, touch screen 2310, joystick 2305, microphone 2306, scanner 2308, and camera device 2307.

[0258] Computer system 2300 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include: tactile output devices (e.g., tactile feedback via touchscreen 2310 or joystick 2305, but tactile feedback devices that are not used as input devices may also exist), audio output devices (e.g., speakers 2309, headphones), and visual output devices (e.g., screens 2310 including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, OLED (Organic Light-emitting Diode) screens, each screen may or may not have touchscreen input capability, each screen may or may not have tactile feedback capability—some of which may be able to output two-dimensional or more three-dimensional visual outputs, for example, using stereoscopic image output, virtual reality glasses, holographic displays and ashtrays, and printers).

[0259] Computer system 2300 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM (Read Only Memory, ROM) / RW (Read Write, RW) 2320 with media such as CD / DVD 2311, thumb drives 2322, removable hard disk drives or solid-state drives 2323, conventional magnetic media such as magnetic tapes and floppy disks, and devices based on dedicated ROM / ASIC (Application Specific Integrated Circuit, ASIC) / PLD (Programmable Logic Device, PLD) such as security dongles.

[0260] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0261] Computer system 2300 may also include interfaces 2399 to one or more communication networks 2398. Network 2398 may be, for example, wireless, wired, or optical. Network 2398 may also be local area, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of network 2398 include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM (Global System for Mobile Communications), 3G (the Third Generation), 4G (the Fourth Generation), 5G (the Fifth Generation), LTE (Long Term Evolution), etc.; cable or wireless wide area digital television networks including cable television, satellite television, and terrestrial broadcast television; and vehicular and industrial networks including CAN (Controller Area Network) buses, etc. Some networks 2398 typically use external network interface adapters that are attached to certain general-purpose data ports or peripheral buses (2350 and 2351) (such as, for example, the USB (Universal Serial Bus, USB) port of computer system 2300); other networks are typically integrated into the core of computer system 2300 by attaching to system buses as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 2398, computer system 2300 can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast television), one-way transmit-only (e.g., a CAN bus to certain CAN bus devices), or bidirectional, such as using a local area digital network or a wide area digital network to other computer systems. Certain protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0262] The human-machine interface devices, human-accessible storage devices, and network interfaces mentioned above can be attached to the core 2340 of the computer system 2300.

[0263] The core 2340 may include one or more central processing units (CPUs) 2341, graphics processing units (GPUs) 2342, graphics adapters 2317, dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) 2343, hardware accelerators 2344 for certain tasks, etc. These devices, along with read-only memory (ROM) 2345, random access memory 2346, and internal mass storage devices such as internal non-user-accessible hard disk drives (SDs), etc. 2347, can be connected via the system bus 2348. In some computer systems, the system bus 2348 can be accessed in the form of one or more physical connectors to allow for expansion via additional CPUs, GPUs, etc. Peripheral devices can be attached directly or via the peripheral bus 2351 to the core's system bus 2348. Peripheral bus architectures include PCI (Peripheral Component Interconnect), USB, etc.

[0264] The CPU 2341, GPU 2342, FPGA 2343, and accelerator 2344 can execute certain instructions, which can be combined to form the aforementioned computer code. This computer code can be stored in ROM 2345 or RAM (Random Access Memory) 2346. Transient data can also be stored in RAM 2346, while permanent data can be stored, for example, in an internal mass storage device 2347. Fast storage and retrieval of any memory device within the memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs 2341, GPUs 2342, mass storage devices 2347, ROM 2345, RAM 2346, etc.

[0265] Computer-readable media may contain computer code for performing various computer-implemented operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or the media and computer code may be of a type known and available to those skilled in the art of computer software.

[0266] By way of example and not limitation, the computer system 2300 with the architecture, and in particular the core 2340, can be functionalized by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, and certain storage devices of the core 2340 with non-transitory characteristics, such as the internal mass storage device 2347 or ROM 2345. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 2340. Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core 2340, and in particular the processors therein (including CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 2346 and modifying such data structures according to the software-defined processes. Alternatively or concurrently, the computer system may provide functionality through hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 2344), which may replace or operate with software to perform the specific processing or a specific portion of the specific processing described herein. Where appropriate, the software mentioned may comprise logic, and conversely, the logic mentioned may comprise software. Where appropriate, the computer-readable medium mentioned may comprise circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0267] While this disclosure has described several exemplary embodiments, there are variations, substitutions, and various alternatives that fall within the scope of this disclosure. Therefore, it will be understood that those skilled in the art will be able to conceive of various systems and methods that, while not expressly shown or described herein, embody the principles of this disclosure and thus fall within its spirit and scope.

Claims

1. A method for video encoding, said method being executed by at least one processor, characterized in that, The method includes: Obtain the input video stream; Based on the input video stream, a neural network is used to generate an encoded video bitstream, the encoded video bitstream including multiple auxiliary enhancement information (SEI) messages and blocks; Determine multiple neural network topology information associated with the neural network; and In a plurality of syntax elements associated with the encoded video bitstream, the determined topology information of a plurality of neural networks is signaled; signaling the determined topology information of a plurality of neural networks includes adaptively selecting one of a picture-level model or a block-level model.

2. The method according to claim 1, characterized in that: Each of the plurality of auxiliary enhancement information (SEI) messages carries one of the determined plurality of neural network topology information; and Multiple auxiliary enhancement information (SEI) messages store images and block-level adaptive information for the multiple neural network topology information.

3. The method according to claim 1, characterized in that, The plurality of syntax elements are notified by signaling via one or more of the plurality of auxiliary enhancement information (SEI) messages, parameter sets, and metadata container boxes.

4. The method according to claim 3, characterized in that: The neural network includes multiple operation nodes, and Generating the encoded video bitstream includes: The input tensor data corresponding to the input video stream is fed to the first operation node in the operation nodes; The input tensor data is processed using either pre-trained constants or variables; and Output intermediate tensor data. The intermediate tensor data includes a weighted sum of the input tensor data and any one of the trained constants and updated variables.

5. The method according to claim 1, characterized in that, The method further includes using an identifier in the Auxiliary Enhancement Information (SEI) message to identify associated neural network topology information among the plurality of neural network topology information for each block of the encoded video bitstream. Different blocks use different neural network topology information from the multiple neural network topology information, and each block includes the identifier and on / off flag.

6. The method according to claim 1, characterized in that: Adaptively selecting the image-level model includes: Segment the image into segments of various sizes; Calculate the minimum cost of the image-level model for each of the segmented images; and Choose the image-level model with the lowest first-rate distortion (RD) cost, and The adaptive selection of the block-level model includes: The block size is compared with one or more block-level models based on the distortion rate of the block; Calculate the minimum cost of the block-level model for each block size; Choose the block-level model with the lowest second-rate distortion (RD) cost, and The method further includes selecting the state of the on / off flag for the selected block-level model and the state of the on / off flag for the selected image-level model based on the first rate-distortion (RD) cost and the second rate-distortion (RD) cost.

7. The method according to any one of claims 1-6, characterized in that, The plurality of neural network topology information is explicitly communicated by signaling using at least one of the following formats: Neural Network Exchange Format (NNEF), Open Neural Network Exchange (ONNX), and MPEG Neural Network Compression Standard (NNR).

8. The method according to claim 7, characterized in that, The neural network is trained using the following methods: A deblocking filter is used to perform deblocking of the encoded video bitstream; A neural network-based reconstruction filter is used to generate a reconstruction output, which is then stored in a buffer. as well as Based on the forward and backward reference images, a neural network-based prediction filter is used to update the reference pixel values ​​of the reconstructed output. The neural network-based reconstruction filter and the neural network-based prediction filter employ the same network model with different training parameters.

9. The method according to claim 8, characterized in that: The neural network-based reconstruction filter is trained by minimizing the error between the reference pixel values ​​of the reconstructed output and the pixel values ​​of the current image; and The neural network-based prediction filter is trained by minimizing the error between the reference pixel values ​​of the reconstructed output and the pixel values ​​of the next image in the decoding order.

10. An apparatus for video encoding, characterized in that, The device includes: At least one memory configured to store program code; and At least one processor is configured to read the program code to execute the method according to any one of claims 1 to 9.

11. A method for video decoding, said method being executed by at least one processor, characterized in that, The method includes: Receive an encoded video bitstream generated using a neural network, the encoded video bitstream including multiple auxiliary enhancement information (SEI) messages and blocks; Receive a plurality of syntax elements associated with the encoded video bitstream, the plurality of syntax elements being used to signal determined plurality of neural network topology information; and Determine multiple neural network topology information associated with the neural network, and signal the determined multiple neural network topology information including adaptively selecting one of the image-level model or the block-level model.

12. A non-transitory computer-readable medium storing instructions, characterized in that, When executed by at least one processor, the instructions cause the at least one processor to perform the method as described in any one of claims 1-10, 11.

13. A method for storing a bit stream, characterized in that, The method of claim 1 is performed to generate a bit stream; and the bit stream is stored.

14. A method for transmitting a bit stream, characterized in that, The method of claim 1 is used to generate a bit stream; and the bit stream is transmitted.

15. A computer-readable storage medium storing a computer program / instructions and a bit stream thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1 to generate the bit stream.

Images