Method, device and storage medium for processing video by neural network

Abstract

Aspects of the disclosure provide methods and apparatus for neural network processing, such as in video processing, for a neural network for video processing including a plurality of channels for processing color components in a color space. In some examples, an apparatus of a neural network processing a video includes processing circuitry. The processing circuitry determines that an input of a convolution operation includes a first input channel that is piecewise constant. The processing circuitry then computes a first intermediate output channel based on other channels of the input of the convolution operation; and then generates an output of the convolution operation based on a combination (e.g., a linear combination) of the first intermediate output channel and the first input channel, for use in inter-picture prediction.

Description

[0001] Cross-references

[0002] This application claims priority to U.S. Patent Application No. 17 / 447,525, filed September 13, 2021, entitled “Method and Apparatus for Video Coding,” which claims priority to U.S. Provisional Application No. 63 / 126,194, filed December 16, 2020, entitled “Application of Partial Depthwise Separable Convolution to Enhance Video Quality in a Video Codec.” The entire disclosure of the prior applications is incorporated herein by reference. Technical Field

[0003] This disclosure describes embodiments that generally involve neural network processing. More specifically, this disclosure provides neural network processing techniques for improving image quality in video codecs. Background Technology

[0004] The background description provided herein is intended to present the overall context of this application. The extent of the work of the currently named inventors described in the background section and various aspects of this specification does not imply that it was prior art at the time of filing of this application, nor is it expressly or implied that it was acknowledged as prior art to this application.

[0005] Video encoding and decoding can be performed using inter-frame prediction techniques with motion compensation. Uncompressed digital video can comprise a series of images, each with spatial dimensions, for example, 1920×1080 luminance samples and correlated chrominance samples. The series of images has a fixed or variable image rate (also informally referred to as the frame rate), such as 60 images per second or 60Hz. Uncompressed video has specific bitrate requirements. For example, a 1080p60 4:2:0 video with 8 bits per sample (1920x1080 luminance sample resolution, 60Hz frame rate) requires close to 1.5 Gbit / s of bandwidth. One hour of such video would require over 600 GB of storage space.

[0006] One objective of video encoding and decoding is to reduce redundant information in the input video signal through compression. Video compression can help reduce the bandwidth and / or storage requirements mentioned above, in some cases by two or more orders of magnitude. Lossless and lossy compression, as well as combinations of both, can be employed. Lossless compression refers to the technique of reconstructing an exact copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal can be used for the intended application. Lossy compression is widely used in video. The amount of distortion tolerated depends on the application. For example, users of some consumer streaming applications may tolerate higher distortion than users of television applications. The achievable compression ratio reflects that a higher allowable / tolerable distortion results in a higher compression ratio.

[0007] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform, quantization, and entropy encoding and decoding.

[0008] Video codec techniques may include techniques called intra-frame coding and decoding. In intra-frame coding and decoding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture can be an intra-frame picture. Intra-frame pictures and their derivatives (such as independent decoder refresh pictures) can be used to reset the decoder state and are therefore used as the first picture in an encoded video stream and video session, or as a still image. Samples of an intra-frame block can be exposed to a transform, and the transform coefficients can be quantized before entropy coding and decoding. Intra-frame prediction can be a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits are needed to represent the entropy-coded block at a given quantization step size.

[0009] Traditional intra-frame encoding and decoding techniques, such as those known from MPEG-2 generation codecs, do not use intra-frame prediction. However, some newer video compression techniques include attempts to use, for example, surrounding sample data and / or metadata, which are obtained during the encoding / decoding of spatially adjacent data blocks and precede the data blocks in decoding order. This technique is hereinafter referred to as "intra-frame prediction." Note that in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and not reference data from a reference picture.

[0010] There can be many different forms of intra-frame prediction. When more than one such technique can be used in a given video codec, the techniques used can be encoded in an intra-frame prediction mode. In some cases, a mode can have sub-modes and / or parameters, and these sub-modes and / or parameters can be encoded separately or included in the mode codeword. The codeword used for a given combination of mode / sub-mode / parameters may affect the coding efficiency gain through intra-frame prediction, as well as the entropy codec techniques that convert the codewords into a bitstream.

[0011] A certain mode of intra-frame prediction was introduced with H.264, improved in H.265, and further refined in newer codec techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied into the predictor block according to the orientation. References to the orientation in use can be encoded in the bitstream or predicted themselves.

[0012] Referring to Figure 1A, the lower right corner depicts a subset of nine known prediction directions from the 33 possible prediction directions of H.265 (corresponding to 33 angular modes of 35 intra-frame modes). The point (101) where the arrows converge represents the sample being predicted. The arrow indicates the direction in which the sample is being predicted. For example, arrow (102) indicates that sample (101) is predicted based on one or more samples at a 45-degree angle to the horizontal direction in the upper right. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more samples at a 22.5-degree angle to the horizontal direction in the lower left.

[0013] Referring again to Figure 1A, a square block (104) comprising 4×4 samples is shown in the upper left (represented by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from top to bottom) and the first sample in the X dimension (from left to right). Similarly, sample S44 is the fourth sample in block (104) in both the X and Y dimensions. Since the block is 4×4 in size, S44 is located in the lower right corner. Reference samples following a similar numbering scheme are also shown. Reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to block (104). In H.264 and H.265, the predicted samples are adjacent to the block being reconstructed, so negative values ​​are not required.

[0014] Intra-frame image prediction can be performed by copying reference sample values ​​from adjacent samples occupied by the prediction direction indicated by the signal. For example, suppose the encoded video bitstream includes signaling that, for this block, the signaling indicates a prediction direction consistent with arrow (102), i.e., predicting samples based on one or more prediction samples at a 45-degree angle to the horizontal direction from the upper right. In this case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is predicted based on reference sample R08.

[0015] In some cases, such as through interpolation, the values ​​of multiple reference samples can be combined to compute a reference sample, especially when the direction is not divisible by 45 degrees.

[0016] With the development of video coding technology, the number of possible directions has increased. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013) and JEM / VVC / BMS, and at the time of this application, up to 65 directions could be supported. Experiments have been conducted to identify the most likely directions, and certain techniques in entropy coding have been used to represent those possible directions using a small number of bits, while accepting some cost for less likely directions. Furthermore, the direction itself can sometimes be predicted based on the adjacent directions used in adjacent, already decoded blocks.

[0017] Figure 1B shows a schematic diagram (105) for depicting 65 intra-frame prediction directions according to JEM, to show the increasing number of prediction directions over time.

[0018] The mapping of intra-predicted direction bits representing direction in an encoded video bitstream can vary depending on the video coding technique, and can range from a simple direct mapping from predicted direction to intra-predicted mode to codewords, to complex adaptive schemes involving the most probable mode, and similar techniques. However, in all cases, there may be certain directions in the video content that are statistically less likely to occur than others. Since the goal of video compression is to reduce redundancy, in well-functioning video codecs, those less likely directions will be represented by a larger number of bits than the more likely directions.

[0019] Motion compensation can be a lossy compression technique and may involve using sample data blocks from a previously reconstructed image or a portion of a reconstructed image (reference image) spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) for prediction of a newly reconstructed image or image portion. In some cases, the reference image may be the same as the image currently being reconstructed. MV may have two dimensions, X and Y, or three dimensions, where the third dimension represents the reference image in use (the latter may indirectly be a temporal dimension).

[0020] In some video compression techniques, the motion vector (MV) applied to a region of sample data can be predicted based on other MVs, such as those MVs associated with another region of sample data adjacent to the region being reconstructed and preceding that MV in the decoding order. This significantly reduces the amount of data required to encode the MV, thereby eliminating redundant information and increasing compression. MV prediction can be performed efficiently, for example, when encoding an input video signal derived from a camera (called natural video), there is a statistical probability that regions larger than the area applicable to a single MV will move in a similar direction. Therefore, in some cases, predictions can be made using similar motion vectors derived from MVs in neighboring regions. This results in MVs found for a given region being similar or identical to MVs predicted from surrounding MVs, and after entropy encoding, can be represented with fewer bits than when directly encoding the MV. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors that arise when calculating predictions based on several surrounding MVs.

[0021] H.265 / HEVC (ITU-T Rec. H.265, “Efficient Video Coding”, December 2016) describes various MV prediction mechanisms. Among the various MV prediction mechanisms provided by H.265, this paper describes a technique referred to below as “spatial combining”.

[0022] Referring to Figure 2, the current block (201) includes samples discovered by the encoder during motion search, which can be predicted based on previous blocks that have been spatially moved by the same size. Instead of directly encoding this MV, it is derived from metadata associated with one or more reference images, such as the most recent (in decoding order) reference image, using the MV associated with any one of the five surrounding samples. The five surrounding samples are denoted as A0, A1 and B0, B1, B2 (from 202 to 206). In H.265, MV prediction can use predictions from the same reference image being used by adjacent blocks. Summary of the Invention

[0023] This disclosure provides methods and apparatus for neural network processing, such as in video processing, where the neural network includes multiple channels for processing color components in a color space. In some examples, the apparatus for processing video via a neural network includes processing circuitry. The processing circuitry determines that the input to a convolution operation includes a piecewise constant first input channel. The processing circuitry then computes a first intermediate output channel based on the other channels of the input to the convolution operation; and then, based on a combination (such as a linear combination) of the first intermediate output channel and the first input channel, generates the output of the convolution operation for use in inter-frame image prediction.

[0024] In some examples, the processing circuitry uses a core to apply a reduced-channel convolution operation to the other input channels to generate a first intermediate output channel.

[0025] In some examples, the processing circuitry can multiply the first input channel by a weight value to generate a second intermediate output channel, and then add the first intermediate output channel to the second intermediate output channel to generate the output of the convolution operation. In one example, the processing circuitry calculates the weight value based on the kernel element corresponding to the first input channel. For example, the processing circuitry calculates the sum of the kernel elements corresponding to the first input channel. In another example, the weight value is pre-calculated based on the kernel element corresponding to the first input channel and stored in memory. The processing circuitry can access the memory containing the stored weight values.

[0026] In some examples, the processing circuitry determines that the first input channel includes a quantization parameter QP mapping for an image received from other input channels. The image is reconstructed based on the QP values ​​in the QP mapping. The processing circuitry then applies a reduced-channel convolution operation to the color components of the image received from the other channels to generate a first intermediate output channel. The processing circuitry generates the output of the convolution operation based on a linear combination of the first intermediate output channel and the QP mapping.

[0027] In some examples, the processing circuitry receives an image from the output of a deblocking filter. In some examples, the processing circuitry may apply an in-loop filter based on a dense residual convolutional neural network to the output of the convolution operation.

[0028] This disclosure also provides a non-volatile computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for processing video using a neural network. Attached Figure Description

[0029] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0030] Figure 1A is a schematic diagram of an exemplary subset of intra-prediction modes.

[0031] Figure 1B is a schematic diagram of an exemplary intra-frame prediction direction.

[0032] Figure 2 is a schematic diagram of an exemplary current block and its surrounding space merging candidates.

[0033] Figure 3 This is a simplified block diagram of a communication system (300) according to an embodiment.

[0034] Figure 4 This is a simplified block diagram of a communication system (400) according to an embodiment.

[0035] Figure 5 This is a simplified block diagram of the decoder according to an embodiment.

[0036] Figure 6 This is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0037] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0038] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0039] Figure 9 Block diagrams of loop filter units in some examples are shown.

[0040] Figure 10 A block diagram of another loop filter unit is shown in some examples.

[0041] Figure 11 Block diagrams of neural network-based filters are shown in some examples.

[0042] Figure 12 Block diagrams of preprocessing modules in some examples are shown.

[0043] Figure 13 Block diagrams of neural network structures in some examples are shown.

[0044] Figure 14 A block diagram of dense residual cells is shown.

[0045] Figure 15 Block diagrams of post-processing modules in some examples are shown.

[0046] Figure 16 A schematic diagram of standard convolution is shown.

[0047] Figure 17 A schematic diagram of depthwise convolution is shown.

[0048] Figure 18 A schematic diagram of point convolution is shown.

[0049] Figure 19 An example of quantization parameter mapping is shown.

[0050] Figure 20 The diagram shows some examples of partially depthwise separable convolution (PDSC).

[0051] Figure 21 A block diagram of a neural network structure is shown.

[0052] Figure 22 A flowchart outlining an example of the process is shown.

[0053] Figure 23 This is a schematic diagram of a computer device according to an embodiment. Detailed Implementation

[0054] According to the embodiments Figure 3 This is a simplified block diagram of a communication system (300) according to an embodiment disclosed in this application. The communication system (300) includes a plurality of terminal devices that can communicate with each other via, for example, a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and terminal device (320) interconnected via a network (350). Figure 3 In this embodiment, terminal device (310) and terminal device (320) perform one-way data transmission. For example, terminal device (310) may encode video data (e.g., a video image stream captured by terminal device (310)) for transmission over a network (350) to another terminal device (320). The encoded video data is transmitted in the form of one or more encoded video streams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover the video data, and display video images based on the recovered video data. One-way data transmission is common in applications such as media services.

[0055] In another embodiment, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, which may occur, for example, during a video conference. For bidirectional data transmission, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video images captured by the terminal device) for transmission over a network (350) to the other terminal device (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other terminal device (330) and (340), and may decode the encoded video data to recover the video data, and may display the video images on an accessible display device based on the recovered video data.

[0056] exist Figure 3 In the embodiments disclosed herein, terminal devices (310), (320), (330), and (340) may be servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) refers to any number of networks, including, for example, wired (connected) and / or wireless communication networks, that transmit encoded video data between terminal devices (310), (320), (330), and (340). The communication network (350) may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of the network (350) may be irrelevant to the operation of this disclosure.

[0057] As an example, Figure 4 The diagram illustrates the placement of a video encoder and a video decoder in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0058] The streaming system may include an acquisition subsystem (413) that may include a video source (401) such as a digital camera, which creates an uncompressed video image stream (402). In an embodiment, the video image stream (402) includes samples captured by a digital camera. The video image stream (402) is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data (404) (or encoded video bitstream). The video image stream (402) may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or carry out aspects of the disclosed subject matter as described in more detail below. Compared to the video image stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower data volume of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on a streaming server (405) for future use. One or more streaming client subsystems, such as Figure 4 Client subsystems (406) and (408) can access a streaming server (405) to retrieve copies (407) and (409) of encoded video data (404). Client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and produces an output video picture stream (411) that can be displayed on a display (412) (e.g., a screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (404), video data (407), and video data (409) (e.g., video streams) may be encoded according to certain video coding / compression standards. Examples of such standards include ITU-T H.265. In embodiments, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and this application can be used in the context of the VVC standard.

[0059] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0060] Figure 5This is a block diagram of a video decoder (510) according to an embodiment disclosed in this application. The video decoder (510) may be disposed in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of... Figure 4 The video decoder (410) in the embodiment.

[0061] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same embodiment or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not indicated). The receiver (531) may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (515) may be located external to the video decoder (510) (not indicated). In other cases, an external buffer (not shown) may be provided for the video decoder (510) to prevent network jitter, for example, and another buffer (515) may be configured internally for, for example, handling broadcast timing. When the receiver (531) receives data from a store / forward device with sufficient bandwidth and controllability or from an isochronous synchronization network, the buffer (515) may not be necessary, or it may be made smaller. Of course, for use on packet networks such as the Internet, a buffer (515) may also be required; this buffer may be relatively large and adaptive in size, and may be at least partially implemented in the operating system or a similar component (not shown) external to the video decoder (510).

[0062] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. These symbols may include information for managing the operation of the video decoder (510) and potential information for controlling a display device (512) (e.g., a display screen), which is not part of the electronic device (530) but may be coupled to it, such as... Figure 5As shown in the diagram. The control information for the display device may be a parameter set fragment (not shown) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / decode the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a subgroup parameter set of at least one subgroup of pixels in the subgroup of pixels for use in the video decoder based on at least one parameter corresponding to a group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0063] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to create symbols (521).

[0064] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (520). For brevity, the flow of such subgroup control information between the parser (520) and the various units described below is not described.

[0065] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0066] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantization transform coefficients as symbols (521) and control information from the parser (520), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

[0067] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses reconstructed information extracted from the current picture buffer (558) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers partially reconstructed and / or fully reconstructed current images. In some cases, the aggregator (555) adds the predictive information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0068] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) can access the reference image memory (557) to extract samples for prediction. After motion compensation is performed on the extracted samples according to the symbols (521), these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (referred to as residual samples or residual signals in this case) to generate output sample information. The motion compensation prediction unit (553) can obtain the predicted samples from the address in the reference image memory (557) under motion vector control, and the motion vector is available to the motion compensation prediction unit (553) in the form of the symbols (521), which, for example, include X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0069] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the encoded video sequence (also referred to as the encoded video stream), and these parameters can be used as symbols (521) from the parser (520) in the loop filter unit (556). However, in other embodiments, the video compression techniques may also respond to metadata obtained during decoding of a previous (in decoding order) portion of the encoded image or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0070] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device (512) and stored in a reference image memory (557) for subsequent inter-frame image prediction.

[0071] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded images corresponding to the current image have been fully reconstructed and the encoded images (by, for example, the parser (520)) are identified as reference images, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0072] The video decoder (510) can perform decoding operations according to a predetermined video compression technique, such as that specified in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference picture size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0073] In this embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0074] Figure 6 This is a block diagram of a video encoder (603) according to an embodiment disclosed in this application. The video encoder (603) is disposed in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace... Figure 4 The video encoder (403) in the embodiment.

[0075] The video encoder (603) can obtain data from the video source (601) (not) Figure 6 In one embodiment, a portion of the electronic device (620) receives video samples, the video source being capable of capturing video images to be encoded by a video encoder (603). In another embodiment, the video source (601) is a portion of the electronic device (620).

[0076] A video source (601) can provide a sequence of source video samples encoded by a video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media service system, the video source (601) may be a storage device storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel may include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0077] According to an embodiment, the video encoder (603) can encode and compress images of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (650) may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used with other suitable functions related to the video encoder (603) optimized for a particular system design.

[0078] In some embodiments, the video encoder (603) operates within an encoding loop. For simplicity, in an embodiment, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded within the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (634) also correspond bit-precisely between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0079] The operation of the “local” decoder (633) can be combined with, for example, the above-mentioned Figure 4 The video decoder (510) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), including the buffer (515) and parser (520), may not be fully implemented in the local decoder (633).

[0080] It can be observed that any decoder technique other than parsing / entropy decoding present in the decoder must also exist in the corresponding encoder in essentially the same functional form. For this reason, this application focuses on decoder operation. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0081] During operation, in some embodiments, the source encoder (630) may perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input image, referencing one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0082] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 When the source video sequence (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0083] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0084] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0085] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0086] The transmitter (640) can buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission via a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).

[0087] The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to any of the following image types:

[0088] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0089] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0090] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0091] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0092] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0093] In this embodiment, the transmitter (640) may transmit additional data while transmitting encoded video. The source encoder (630) may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, SEI messages, VUI parameter set fragments, etc.

[0094] The acquired video can serve as multiple source images (video images) presented in a time series. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0095] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. Specifically, the block can be predicted using a combination of the first and second reference blocks.

[0096] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0097] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In embodiments, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0098] Figure 7 This is a diagram of a video encoder (703) according to another embodiment disclosed in this application. The video encoder (703) is used to receive processing blocks (e.g., prediction blocks) of sample values ​​within a current video image in a video image sequence, and to encode the processing blocks into an encoded image that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used instead of Figure 4The video encoder (403) in the embodiment.

[0099] In the HEVC embodiment, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8×8 sample prediction block. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to use an intra-frame mode, an inter-frame mode, or a bidirectional prediction mode to encode the processing block. When encoding the processing block in intra-frame mode, the video encoder (703) can use intra-frame prediction techniques to encode the processing block into an already encoded picture; and when encoding the processing block in inter-frame mode or bidirectional prediction mode, the video encoder (703) can use inter-frame prediction or bidirectional prediction techniques to encode the processing block into an already encoded picture, respectively. In some video coding techniques, the merging mode can be an inter-frame picture prediction sub-mode, in which motion vectors are derived from one or more motion vector prediction values ​​without relying on already encoded motion vector components outside the prediction values. In some other video coding techniques, motion vector components applicable to the subject block may exist. In the embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the processing block mode.

[0100] exist Figure 7 In one embodiment, the video encoder (703) includes, as shown below: Figure 7 The inter-frame encoder (730), intra-frame encoder (722), residual calculator (723), switch (726), residual encoder (724), general controller (721) and entropy encoder (725) are shown coupled together.

[0101] An inter-frame encoder (730) is configured to receive samples of the current block (e.g., the processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in previous and later images), generate inter-frame prediction information (e.g., redundancy information description, motion vectors, merging mode information based on inter-frame coding techniques), and calculate inter-frame prediction results (e.g., predicted blocks) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image based on encoded video information.

[0102] The intra encoder (722) is used to receive samples of the current block (e.g., the processing block), in some cases compare the block with previously encoded blocks in the same image, generate quantization coefficients after transformation, and in some cases also (e.g., based on intra prediction direction information of one or more intra coding techniques) generate intra prediction information. In an embodiment, the intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0103] A general controller (721) determines general control data and controls other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the mode of a block and provides control signals to a switch (726) based on the mode. For example, when the mode is an intra-frame mode, the general controller (721) controls the switch (726) to select an intra-frame mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-frame prediction information and add the intra-frame prediction information to the bitstream; and when the mode is an inter-frame mode, the general controller (721) controls the switch (726) to select an inter-frame prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-frame prediction information and add the inter-frame prediction information to the bitstream.

[0104] A residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). A residual encoder (724) is used to operate on the residual data to encode the residual data to generate transform coefficients. In an embodiment, the residual encoder (724) is used to transform the residual data from the time domain to the frequency domain and generate transform coefficients. The transform coefficients are then quantized to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter-frame prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra-frame prediction information. The decoded blocks are processed appropriately to generate a decoded image, and in some embodiments, the decoded image may be buffered in a memory circuit (not shown) and used as a reference image.

[0105] An entropy encoder (725) is used to format the bitstream to produce encoded blocks. The entropy encoder (725) generates various information according to a suitable standard such as the HEVC standard. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (e.g., intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the bitstream. It should be noted that, according to the disclosed subject matter, residual information is not present when blocks are encoded in a merged sub-mode of inter-frame mode or bidirectional prediction mode.

[0106] Figure 8This is a diagram of a video decoder (810) according to another embodiment disclosed in this application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is used instead of Figure 4 The video decoder (410) in the embodiment.

[0107] exist Figure 8 In the embodiment, the video decoder (810) includes, as follows: Figure 8 The entropy decoder (871), inter-frame decoder (880), residual decoder (873), reconstruction module (874), and intra-frame decoder (872) are shown coupled together.

[0108] An entropy decoder (871) can be used to reconstruct certain symbols from an encoded image, these symbols representing the syntax elements constituting the encoded image. Such symbols may include, for example, a mode for encoding the block (e.g., intra-frame mode, inter-frame mode, bidirectional prediction mode, a combined sub-mode of the latter two, or another sub-mode), prediction information (e.g., intra-frame prediction information or inter-frame prediction information) that can respectively identify certain samples or metadata used by the intra-frame decoder (872) or the inter-frame decoder (880) for prediction, residual information in the form of, for example, quantized transform coefficients, and so on. In an embodiment, when the prediction mode is inter-frame or bidirectional prediction mode, inter-frame prediction information is provided to the inter-frame decoder (880); and when the prediction type is intra-frame prediction type, intra-frame prediction information is provided to the intra-frame decoder (872). Residual information may be provided to the residual decoder (873) via inverse quantization.

[0109] The inter-frame decoder (880) is used to receive inter-frame prediction information and generate inter-frame prediction results based on the inter-frame prediction information.

[0110] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate prediction results based on the intra-frame prediction information.

[0111] The residual decoder (873) performs inverse quantization to extract the dequantized transform coefficients and processes the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also require some control information (to obtain the quantizer parameters QP), and this information may be provided by the entropy decoder (871) (the data path is not indicated because this is only low-level control information).

[0112] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which may be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, which may be a part of a reconstructed image, which in turn may be a part of a reconstructed video. It should be noted that other suitable operations, such as deblocking, may be performed to improve visual quality.

[0113] It should be noted that any suitable technology can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In one embodiment, one or more integrated circuits can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810). In another embodiment, one or more processors executing software instructions can be used to implement the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810).

[0114] Neural network technology can be used in conjunction with video encoding and decoding technology, and video encoding and decoding technology with neural networks can be referred to as hybrid video encoding and decoding technology. For example, a loop filter unit such as a loop filter unit (556) can apply various loop filters to perform sample filtering. One or more loop filters can be implemented by a neural network. Aspects of this disclosure provide in-loop filtering techniques in hybrid video encoding and decoding technology that improve image quality by using neural networks. Specifically, according to one aspect of this disclosure, partially depthwise separable convolution techniques can be used in video encoding and decoding to enhance video quality.

[0115] According to some aspects of this disclosure, adaptive filtering in hybrid video encoding and decoding techniques can use a neural network with quantization parameter mapping as one of the inputs for improving image quality. To reduce computation and the number of weight parameters in the neural network, a type of convolutional neural network layer called partially depthwise separable convolution (PDSC) can be used.

[0116] In some examples (e.g., JVET-T0057), normal convolution and depthwise separable convolution (DSC) are used in the in-loop filter.

[0117] According to one aspect of this disclosure, an in-loop filter is a filter that affects reference data. For example, an image filtered by the loop filter unit (556) is stored in a buffer such as a reference image memory (557) as a reference for further prediction. In-loop filters can improve video quality in a video codec.

[0118] Figure 9 Block diagrams of loop filter units (900) in some examples are shown. In these examples, loop filter unit (900) can be used instead of loop filter unit (556). Figure 9 In the example, the loop filter unit (900) includes a deblocking filter (901), a sample adaptive offset (SAO) filter (902), and an adaptive loop filter (ALF) filter (903). In some examples, the ALF filter (903) may include an adaptive loop filter (CCALF).

[0119] During operation, in the example, the loop filter unit (900) receives the reconstructed image, applies various filters to the reconstructed image, and generates an output image in response to the reconstructed image.

[0120] In some examples, the deblocking filter (901) and the SAO filter (902) are configured to remove block artifacts and sample distortion introduced when using block coding techniques. The deblocking filter (901) can smooth shape edges formed when using block coding techniques. The SAO filter (902) can apply a specific offset to a sample to reduce distortion relative to other samples in the video frame. The ALF (903) can apply a classification to, for example, a sample block, and then apply a filter associated with that classification to the sample block. In some examples, the filter coefficients can be determined by the encoder and signaled to the decoder.

[0121] In some examples (e.g., JVET-T0057), an additional filter called a dense residual convolutional neural network based in-loop filter (DRNLF) can be inserted between the deblocking filter (901) and the SAO filter (902). DRNLF can further improve image quality.

[0122] Figure 10Block diagrams of loop filter units (1000) in some examples are shown. In these examples, loop filter unit (1000) can be used in place of loop filter unit (556). Figure 10 In the example, the loop filter unit (1000) includes a deblocking filter (1001), a SAO filter (1002), an ALF filter (1003), and a DRNLF filter (1010) placed between the deblocking filter (1001) and the SAO filter (1002).

[0123] The deblocking filter (1001) is similarly configured as the deblocking filter (901), the SAO filter (1002) is similarly configured as the SAO filter (902), and the ALF filter (1003) is similarly configured as the ALF filter (903).

[0124] The DRNLF filter (1010) receives the output of the deblocking filter (1001) shown by the deblocked image (1011), and also receives a quantization parameter (QP) mapping of the reconstructed image. The QP mapping includes the quantization parameters of the blocks in the reconstructed image. The DRNLF filter (1010) can output an image shown by a filtered image (1019) with improved quality, and the filtered image (1019) is fed to the SAO filter (1002) for further filtering.

[0125] According to one aspect of this disclosure, a neural network for video processing may include multiple channels for processing color components in a color space. In an example, the color space may be defined using the YCbCr model. In the YCbCr model, Y represents the luminance component (luminance), and Cb and Cr represent the chrominance components. It should be noted that in the following description, YUV is used to describe a format encoded using the YCbCr model.

[0126] According to one aspect of this disclosure, multiple channels in a neural network are configured to operate on color components of the same size. In some examples, an image can be represented by color components of different sizes. For example, the human visual system is much more sensitive to changes in brightness than to changes in color; therefore, video systems can compress chroma components to reduce file size and save transmission time without the large visual differences perceived by the human eye. In some examples, chroma subsampling techniques utilize the human visual system's sensitivity to color differences rather than brightness information to achieve a lower resolution for chroma information than for brightness information.

[0127] In some examples, subsampling can be represented as a three-part ratio, such as 4:4:4, 4:2:0, 4:2:2, 4:1:1, etc. For example, 4:4:4 (also known as YUV444) indicates that each YCbCr component has the same sampling rate without subsampling; 4:2:0 (also known as YUV420) indicates that the chrominance components are subsampled, with each four pixels (or Y component) corresponding to the Cb and Cr components. It should be noted that YUV420 is used in the following description as an example of a subsampling format to illustrate the techniques in this disclosure. The techniques can be used for other subsampling formats.

[0128] Typically, neural networks can operate on images in non-subsampled formats (e.g., YUV444). Therefore, for subsampled images, the image is converted to a non-subsampled format before being fed into the neural network.

[0129] Figure 11 A block diagram of a DRNLF filter (1100) in some examples is shown. In the examples, the DRNLF filter (1100) can be used instead of the DRNLF filter (1010). The DRNLF filter (1100) includes, as follows: Figure 11 The QP mapping quantizer (1110), preprocessing module (1120), main processing module (1130), and postprocessing module (1140) are coupled together as shown. The main processing module (1130) includes, as... Figure 11 The patch acquirer (1131), the patch-based DRNLF kernel processing module (1132), and the patch reassembler (1133) are shown coupled together.

[0130] In some examples, the QP mapping includes a mapping of QP values ​​that is applied to reconstruct individual blocks in the currently reconstructed image. The QP mapping quantizer (1110) can quantize the values ​​into a predetermined set of values. In an example (e.g., JVET-T0057), the QP values ​​can be quantized by the QP mapping quantizer (1110) to one of 22, 27, 32, and 37.

[0131] The preprocessing module (1120) can receive a deblocked picture in a first format and convert it to a second format used by the main processing module (1130). For example, the main processing module (1130) is configured to process pictures in YUV444 format. When the preprocessing module (1120) receives a deblocked picture in a different format than YUV444, the preprocessing module (1120) can process the deblocked picture in the different format and output a deblocked picture in YUV444 format. For example, the preprocessing module (1120) receives a deblocked picture in YUV420 format and then interpolates the U chroma channel and V chroma channel horizontally and vertically by a factor of 2 to generate a deblocked picture in YUV444 format.

[0132] The main processing module (1130) can receive a deblocked image in YUV444 format and a quantized QP map as input. The patch extractor (1131) decomposes the input into patches. The DRNLF kernel processing module (1132) can process each patch in the patch separately based on the DRNLF kernel. The patch reassembler (1133) can assemble the patches processed by the DRNLF kernel processing module (1132) into a filtered image in YUV444 format.

[0133] The post-processing module (1140) converts the filtered image in the second format back to the first format. For example, the post-processing module (1140) receives a filtered image in YUV444 format (output from the main processing module (1130)) and outputs a filtered image in YUV420 format.

[0134] Figure 12 A block diagram of a preprocessing module (1220) is shown in some examples. In the examples, the preprocessing module (1220) is used instead of the preprocessing module (1120).

[0135] The preprocessing module (1220) can receive a deblocked image in YUV420 format, convert the deblocked image into YUV444 format, and output a deblocked image in YUV444 format. Specifically, the preprocessing module (1220) receives the deblocked image in three input channels, including a luminance input channel for the Y component and two chrominance input channels for the U(Cb) component and V(Cr) component, respectively. The preprocessing module (1220) outputs the deblocked image through three output channels, including a luminance output channel for the Y component and two chrominance output channels for the U(Cb) component and V(Cr) component, respectively.

[0136] In the example, when the de-patch image has a YUV420 format, the Y component has a size (H, W), the U component has a size (H / 2, W / 2), and the V component has a size (H / 2, W / 2), where H represents the height of the de-patch image (e.g., in samples) and W represents the width of the de-patch image (e.g., in samples).

[0137] exist Figure 12 In the example, the preprocessing module (1220) does not adjust the size of the Y component. The preprocessing module (1220) receives a Y component of size (H, W) from the luminance input channel and outputs a Y component of size (H, W) to the luminance output channel.

[0138] The preprocessing module (1220) adjusts the sizes of the U and V components, respectively. The preprocessing module (1220) includes a first resizing unit (1221) and a second resizing unit (1222) that process the U and V components respectively. For example, the first resizing unit (1221) receives a U component of size (H / 2, W / 2), resizes the U component to size (H, W), and outputs the U component of size (H, W) to the chroma output channel of the U component. The second resizing unit (1222) receives a V component of size (H / 2, W / 2), resizes the V component to size (H, W), and outputs the V component of size (H, W) to the chroma output channel for the V component. In some examples, the first resizing unit (1221) adjusts the size of the U component based on interpolation, for example, using a Lanczos interpolation filter. Similarly, in some examples, the second resizing unit (1222) uses, for example, a Lanczos interpolation filter to resize the V component based on interpolation.

[0139] In some examples, interpolation operations, such as those using Lanczos interpolation filters, do not guarantee that the output of the interpolation operation will be a meaningful value; for example, the U(Cb) and V(Cr) components must be non-negative for meaningful interpolation. In some examples, the output of the interpolation operation can be clipped to a non-negative value. In some examples, preprocessed deblocked images in YUV444 format can be stored and then used during the training of the neural network. Negative values ​​of the U(Cb) and V(Cr) components can adversely affect the results of the neural network training process.

[0140] Figure 13A block diagram of the neural network architecture (1300) is shown. In some examples, the neural network architecture (1300) is used for a dense residual convolutional neural network based in-loop filter (DRNLF) and can be used to replace the patch-based DRNLF kernel processing module (1132). The neural network architecture (1300) includes a series of dense residual units (DRUs), such as DRU(1301)-DRU(1304), and the number of DRUs is represented by N. Figure 13 In this context, the number of convolution kernels is represented by M, which is also the number of output channels used for convolution. For example, "CONV 3×3×M" indicates a standard convolution with M kernels of size 3×3, and "DSC 3×3×M" indicates a depthwise separable convolution with M kernels of size 3×3. N and M can be set according to the trade-off between computational efficiency and performance. In the example (e.g., JVET-T0057), N is set to 4, and M is set to 32.

[0141] During operation, the neural network structure (1300) processes the deblocked image through patches. For each patch of the deblocked image in YUV444 format, the patch is normalized (e.g., in...). Figure 13 In the example, the average value of the deblocked image is removed from the normalized patch (divided by 1023) to obtain the first part (1311) of the internal input (1313). The second part of the internal input (1313) comes from the QP map. For example, a patch of the QP map corresponding to the patch forming the first part (1311) is obtained from the QP map (referred to as the QP map patch). The QP map patch is normalized (e.g., in the example, divided by 1023), and the average value of the deblocked image is removed from the normalized patch to obtain the first part (1311) of the internal input (1311). Figure 13 Divide by 51). The normalized QP mapping patch is the second part (1312) of the internal input (1313). The second part (1312) is concatenated with the first part (1311) to obtain the internal input (1313). The internal input (1313) is provided to the first regular convolutional block (1351) (represented by CONV 3×3×M). The output of the first regular convolutional block (1351) is then processed by N DRUs. It should be noted that the output of the first regular convolutional block (1351) includes M channels.

[0142] For each DRU, an intermediate input is received and processed. The output of the DRU is concatenated with the intermediate input to form the intermediate input for the next DRU. Using DRU (1302) as an example, DRU (1302) receives intermediate input (1321), processes intermediate input (1321), and generates output (1322). Output (1322) is concatenated with intermediate input (1321) to form the intermediate input (1323) for DRU (1303).

[0143] It should be noted that, since the intermediate input (1321) has more than M channels, the "CONV 1×1×M" convolution operation can be applied to the intermediate input (1321) to generate M channels for further processing by the DRU (1302). It should also be noted that the output of the first regular convolution block (1351) includes M channels, and therefore this output can be processed by the DRU (1301) without using the "CONV 1×1×M" convolution operation.

[0144] The output of the final DRU is fed into the final regular convolutional block (1359). For example, by... Figure 13 The method shown here adds the average value of the de-patch images and multiplies it by 1023, converting the output of the final regular convolutional block (1359) into regular image patch values.

[0145] Figure 14 A block diagram of a dense residual element (DRU) (1400) is shown. In some examples, the DRU (1400) can be used instead of... Figure 13 Each DRU in the dataset (such as DRU(1301), DRU(1302), DRU(1303), and DRU(1304)).

[0146] exist Figure 14 In the example, the DRU (1400) receives an intermediate input x and propagates it directly to subsequent DRUs via a shortcut (1401). The DRU (1400) also includes a regular processing path (1402). In some examples, the regular processing path (1402) includes a regular convolutional layer (1411), depthwise separable convolution (DSC) layers (1412) and (1414), and a rectified linear unit (ReLU) layer (1413). For example, the intermediate input x is concatenated with the output of the regular processing path (1402) to form an intermediate input for subsequent DRUs.

[0147] In some examples, DSC layers (1412) and (1414) are used to reduce computational costs.

[0148] According to one aspect of this disclosure, the neural network structure (1300) includes three channels corresponding to the Y component, U (Cb) component, and V (Cr) component, respectively. In some examples, these three channels may be referred to as the Y channel, U channel, and V channel. The DRNLF filter (1100) can be applied to intra-frame and inter-frame images. In some examples, additional flags are used to indicate the on / off state of the DRNLF filter (1100) at the image level and CTU level.

[0149] Figure 15 A block diagram of a post-processing module (1540) is shown in some examples. In the examples, post-processing module (1540) can be used in place of post-processing module (1140). Post-processing module (1540) includes limiting units (1541)-(1543), which limit the values ​​of the Y component, U component, and V component to a predetermined non-negative range [a, b], respectively. In the examples, the lower limit a and upper limit b of the non-negative range can be set to a = 16 × 4 and b = 234 × 4, respectively. Further, post-processing module (1540) includes resizing units (1545) and (1546), which resize the sizes of the limited U component and V component from size (H, W) to size (H / 2, W / 2), respectively, where H is the height of the original image (e.g., a deblocked image) and W is the width of the original image.

[0150] In some examples (e.g., JVET-T0057), the in-loop filter can use standard convolution, depthwise convolution, and depthwise separable convolution (DSC). Standard convolution, depthwise convolution, and depthwise separable convolution (DSC) will be described below.

[0151] In some examples, standard convolution (also known as regular convolution or normal convolution) can be applied directly in both spatial and depth dimensions.

[0152] Figure 16 A schematic diagram of standard convolution is shown. Figure 16 In the example, the input tensor (1610) corresponding to the block (e.g., an 8×8 patch) can include a width dimension, a height dimension, and a depth dimension. For example, the width is the width of the block, the height is the height of the block, and the depth includes four input channels, such as three input channels for the Y, Cb, and Cr components of the block and one input channel for the QP mapping of the block. When the block is an 8×8 block (also known as a patch), the input tensor (1610) is a 4×8×8 array. The filter (1620) is a 4×3×3 kernel. In standard convolution, for each output channel, for example... Figure 16In the output channel (1630), standard convolution can directly apply convolution to the input tensor (1610) and the filter (1620) in the depth, width, and height dimensions to generate an 8×8 array of output channels (1630). Standard convolution performs depth and spatial computation in one step.

[0153] In some examples such as PyTorch (an open-source machine learning library), a standard convolution can be computed by applying a 2D convolution to an input signal consisting of several input patches, which generates an output signal. The input signal can be an image frame and can consist of elements of size (N, C). in The input tensor (H, W) is used to represent the number of patches, also known as the batch size, where N represents the number of patches and is also called the batch size. in The input signal represents the number of input channels (e.g., depth), H represents the height of the image frame (in pixels), and W represents the width of the image frame (in pixels). The output signal can be a variable with a value of (N, C). out H out W out The output tensor representation of C) out H represents the number of output channels. out This indicates the height of the output image, and W... out This indicates the width of the output image. In some examples, H... out Equal to H, and W out It equals W. In the example, the output signal can be described according to equation (1).

[0154]

[0155] Where, N i An index representing a patch (also known as a batch). Indicates the index of the output channel. This indicates the bias value of the output channel; * indicates the 2-D cross-correlation operator. It can be the sum of the k-th input channel and The 2D weighted array (spatial dimension) of the kernel associated with the output channel, input(N) i (k) represents the patch N used in the input signal. i The 2-D array (spatial dimension) of the k-th input channel.

[0156] It should be noted that, using standard convolution, the number of weight parameters involved in the neural network is approximately C. in ×C out ×H f ×W f H f W represents the height of the filter.f This represents the width of the filter. In the example, the number of input channels is 4, the number of output channels is 32, the filter height is 3, and the filter width is 3, so the number of weight parameters is 1152.

[0157] Depthwise convolution refers to performing a convolution between each filter channel and its corresponding input channel.

[0158] Figure 17 A schematic diagram of depthwise convolution is shown. Figure 17 In the example, the input tensor (1710) is similar to the input tensor (1610). Based on the depth dimension, the input tensor (1710) comprises four input channels (1711)-(1714), and each input channel comprises a 2D array in the spatial dimension. Then, four filter channels (1721)-(1724) corresponding to the input channels (1711)-(1714) are used for depthwise convolution.

[0159] For depthwise convolution, a first convolution is applied to the input channel (1711) and the filter channel (1721) to generate the output channel (1731); a second convolution is applied to the input channel (1712) and the filter channel (1722) to generate the output channel (1732); a third convolution is applied to the input channel (1713) and the filter channel (1723) to generate the output channel (1733); and a fourth convolution is applied to the input channel (1714) and the filter channel (1724) to generate the output channel (1734). The output channels (1731)-(1734) are stacked into an output tensor (1730).

[0160] Depthwise separable convolution (DSC) performs depth and spatial computation through a first step of depthwise convolution and a second step of pointwise convolution. (Reference) Figure 17 Depth convolution is shown and described, and the output tensor (1730) can be referred to as the intermediate output tensor by the first step of DSC.

[0161] Figure 18 A schematic diagram of point convolution that can be used in the second step of DSC is shown. Point convolution can be applied to the channels of an intermediate output tensor (1830) with points (1840) (e.g., corresponding to the output tensor (1730)) to generate an output channel (1850) from the depth convolution as a linear combination of the channels of the intermediate output tensor (1830).

[0162] For DSC-based computation, the input signal can be an image frame, and can be of size (N, C). in The input tensor (H,W) is used to represent the number of patches, also known as the batch size, where N represents the number of patches and is also called the batch size. inThe input signal represents the number of input channels (e.g., depth), H represents the height of the image frame (in pixels), and W represents the width of the image frame (in pixels). The output signal can be a variable with a value of (N, C). out H out W out The output tensor representation of C) out H represents the number of output channels. out This indicates the height of the output image, and W... out This indicates the width of the output image. In some examples, H... out Equal to H, and W out Equal to W. In some examples, for each channel (input channel k), the calculation of the first step of DSC can be expressed by equation (2):

[0163]

[0164] Where, N i This represents the index of the patch (also known as a batch), and k represents the index of the channel. Indicates the channel bias; ★ indicates the 2D cross-correlation operator. It can be a 2D weighted array (spatial dimension) of the kernel of the k-th channel, where input(Ni, k) represents patch N in the input signal. i The 2D array (spatial dimension) of the k-th channel, It can represent patch N i The intermediate output of the kth channel.

[0165] In some examples, the pointwise convolution in the second step can generate the output of the DSC as a linear combination of the intermediate outputs of the depthwise convolution, as exemplified by equation (3):

[0166]

[0167] Where, N i An index representing a patch (also known as a batch). Indicates the index of the output channel. This indicates the bias value of the output channel; ★ indicates the multiplication operator. It is possible to configure the k-th channel and the output channel. Weighted values ​​associated with each other Indicates patch N i The intermediate output of the kth channel.

[0168] It should be noted that when using depthwise separable convolutions, the number of weight parameters (also known as neural network parameters, model parameters, filter kernel parameters, etc.) involved in the neural network is approximately C. in ×H f ×Wf +C in ×C out , where H f W represents the height of the filter. f This represents the width of the filter. In the example, the number of input channels is 4, the number of output channels is 32, the filter height is 3, and the filter width is 3, so the number of weight parameters is 164. It should also be noted that using depthwise separable convolution reduces computation compared to standard convolution.

[0169] According to some aspects of this disclosure, when a quantization parameter (QP) mapping is used as one of the input channels of a neural network-based loop filter, the computation and the number of network parameters in the neural network-based loop filter can be further reduced by using a neural network layer structure known as partial depthwise separable convolution (PDSC).

[0170] According to one aspect of this disclosure, the QP mapping can be piecewise constant, which includes local constant values ​​such as those within each coding block. For example, the input channel is piecewise constant when the input channel is spatially divided into blocks (e.g., blocks larger than the size of the convolution kernel (the width and height of the block are greater than the width and height of the convolution kernel)) and the pixels in each block have the same value.

[0171] Figure 19 An example of a QP mapping for image (1900) is shown. In this example, the QP mapping is formed by the QP values ​​of the pixels in image (1900). Then, pixels in the encoding unit have the same QP value. Figure 19 In this image (1900), the image can be divided into multiple coding units, such as a first coding unit (1901), a second coding unit (1902), etc. Therefore, pixels in the first coding unit (1901) have a first QP value QP1, and pixels in the second coding unit (1902) have a second QP value QP2. The QP mapping of the image is segment-constant, and the QP mapping of the coding units is constant.

[0172] According to one aspect of this disclosure, convolution with segmented constant channels can be approximated by multiplication operations to reduce the number of computations and model parameters.

[0173] Using standard convolution as an example, which includes QP mappings as input channels, equation (1) can be rewritten as equation (4):

[0174]

[0175]

[0176] When, for example, in the case of QP mapping, input channel C in -1 represents the piecewise constant value, and is related to input(N) i C in The convolution of -1) can be approximated by multiplication to reduce the computation and the number of model parameters, as shown in equation (5):

[0177]

[0178] In some examples, yes The sum of the (nuclear) elements, the It is related to input channel C in -1 and output channels The associated 2D weight matrix. The neural network layer structure according to equation (5) is called partially depthwise separable convolution (PDSC). It should be noted that equation (5) uses multiplication operations instead of convolution operations in equation (4), thus reducing computation and the number of model parameters.

[0179] Figure 20 Illustrations of partially depthwise separable convolutions (PDSC) are shown in some examples. Figure 20 In the example, the input tensor (2010) is similar to the input tensor (1610). The input tensor (2010) is decomposed into an intermediate input tensor (2011) and a piecewise constant input channel (2012). Then, a convolution operation (with a reduced number of input channels) is applied to the intermediate input tensor (2011) and the filter (2021) to generate a first intermediate output channel (2031). A linear combination of the first intermediate output channel (2031) and the piecewise constant input channel (2012) can then be performed to generate an output channel (2030). For example, multiplying the weight value (2022) with the piecewise constant input channel (2012) generates a second intermediate output channel (2032). Adding the first intermediate output channel (2031) and the second intermediate output channel (2032) generates a partially depthwise separable convolutional output channel (2030).

[0180] According to some aspects of this disclosure, partially depth-separable convolutions can be used for any suitable convolution, which includes segmented constant input channels.

[0181] In the example, the QP map is used as one of the input channels of the first regular convolutional block (1351), and then the first regular convolutional block (1351) can be modified to use partial depth-separable convolution.

[0182] Figure 21 A block diagram of the neural network structure (2100) is shown. In some examples, the neural network structure (2100) is used for a dense residual convolutional neural network based in-loop filter (DRNLF) and can be used in place of the patch-based DRNLF kernel processing module (1132).

[0183] The neural network structure (2100) is similarly configured as the neural network structure (1300) and utilizes some components that are the same as or equivalent to those used in the neural network structure (1300). For example, the neural network structure (2100) includes a series of dense residual units (DRUs) (e.g., DRU(2101)-DRU(2104)), and the number of DRUs is represented by N. DRU(2101)-DRU(2104) is similarly configured as DRU(1301)-DRU(1304). The neural network structure (2100) includes a final regular convolutional block (2159) configured similarly to the final regular convolutional block (1359). Descriptions of these components have been provided above, and for the sake of brevity, these descriptions will be omitted here.

[0184] However, the neural network architecture (2100) includes a first convolutional block (2151) configured differently from the first conventional convolutional block (1351). Specifically, implementing the first convolutional block (2151) using partially depth-separable convolution (PDSC) can reduce computation and the number of network parameters.

[0185] exist Figure 21 In the example, the internal input (2113) of the first convolutional block (2151) includes a piecewise constant QP mapping. Specifically, the internal input (2113) of the first convolutional block (2151) includes a first part (2111) and a second part (2112). In the example, the first part (2111) is generated from a deblocked reconstructed image, which may include three components (also referred to as three channels), such as a Y channel, a Cb channel, and a Cr channel. The second part (2112) is a normalized QP mapping. The second part (2112) is concatenated with the first part (2111) to obtain the internal input (2113). The internal input (2113) is provided to the first convolutional block (2151).

[0186] According to one aspect of this disclosure, the (normalized) QP mapping is piecewise constant, and the first convolutional block (2151) can be determined according to equation (5) and / or Figure 20 This is achieved by reducing computation and network parameters.

[0187] Figure 22 A flowchart outlining a process (2200) according to an embodiment of the present disclosure is shown. Process (2200) can be used in neural network processing, such as in a neural network-based in-loop filter in a video codec. In various embodiments, process (2200) is executed by processing circuitry, such as processing circuitry in terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (403), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), and processing circuitry performing the functions of a video encoder (603), etc. In some embodiments, process (2200) is implemented as software instructions, so that when the processing circuitry executes the software instructions, the processing circuitry executes process (2200). The process begins at (S2201) and proceeds to (S2210).

[0188] At (S2210), the input to the convolution operation is separated into a first input channel that includes segmented constant.

[0189] At (S2220), a first intermediate output channel is computed based on the other channels of the input from the convolution operation. For example, a reduced-channel convolution operation is applied to the other channels of the input using a kernel to generate the first intermediate output channel.

[0190] At (S2230), the output of the convolution operation is generated based on a combination (e.g., a linear combination) of the first intermediate output channel and the first input channel. Then, the process proceeds to (S2299).

[0191] In some examples, the first input channel is multiplied by a weight value to generate a second intermediate output channel. Then, the first intermediate output channel is added to the second intermediate output channel to generate the output of the convolution operation.

[0192] In the example, the weight values ​​are calculated based on the kernel elements corresponding to the first input channel. For example, the weight value is calculated as the sum of the kernel elements corresponding to the first input channel. In some examples, the weight values ​​are pre-calculated based on the kernel elements corresponding to the first input channel and then stored.

[0193] In the example, the weights are one of the model parameters of the neural network, and the weights are determined based on the training data using other model parameters.

[0194] In some examples, the first input channel includes a quantization parameter (QP) mapping for an image received from the other input channels. The image is reconstructed based on the QP values ​​in the QP mapping. Then, a reduced-channel convolution operation is applied to the color components of the image received from the other channels through a kernel to generate a first intermediate output channel. The output of the convolution operation is generated based on a linear combination of the first intermediate output channel and the QP mapping. In some examples, the image is output from a deblocking filter. In some examples, an in-loop filter based on a dense residual convolutional neural network may be applied to the output of the convolution operation.

[0195] It should be noted that the various units, blocks, and modules described above can be implemented using various technologies (such as processing circuits, processors that execute software instructions, combinations of hardware and software, etc.).

[0196] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 23 A computer device (2300) is shown, which is adapted to implement certain embodiments of the disclosed subject matter.

[0197] The computer software can be encoded using any suitable machine code or computer language, and code including instructions can be created through mechanisms such as assembly, compilation, and linking. These instructions can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or executed through decoding, microcode, etc.

[0198] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0199] Figure 23 The components shown for the computer device (2300) are exemplary in nature and are not intended to limit the scope or functionality of the computer software used to implement the embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any component or combination thereof shown in the exemplary embodiments of the computer device (2300).

[0200] The computer device (2300) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, swiping, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-machine interface device may also be used to capture certain media, which need not be directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0201] Human-machine interface input devices may include one or more of the following (only one is shown): keyboard (2301), mouse (2302), touchpad (2303), touch screen (2310), data glove (not shown), joystick (2305), microphone (2306), scanner (2307), camera (2308).

[0202] The computer device (2300) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (2310), data gloves (not shown), or joystick (2305), but may also include tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (2309), headphones (not shown)), visual output devices (e.g., screens (2310) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light-emitting diode screens, each of which may or may not have touchscreen input functionality, each of which may or may not have tactile feedback functionality—some of which may output two-dimensional or more three-dimensional visual outputs by means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke boxes (not shown)), and printers (not shown).

[0203] Computer equipment (2300) may also include human-accessible storage devices and related media, such as optical media including high-density read-only / rewritable optical discs (CD / DVD ROM / RW) (2320) or similar media (2321) with CD / DVD, thumb drives (2322), removable hard disk drives or solid-state drives (2323), conventional magnetic media such as magnetic tapes and floppy disks (not shown), dedicated devices based on ROM / ASIC / PLD such as security software protectors (not shown), etc.

[0204] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0205] The computer device (2300) may also include an interface (2355) to one or more communication networks (2354). The network may be wireless, wired, or optical. The network may also be a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), vehicular and industrial network, real-time network, latency-tolerant network, etc. Examples of networks may include Ethernet, wireless LAN, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), cable or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicular and industrial networks (including CANBus), etc. Some networks typically require an external network interface adapter for connection to certain general-purpose data ports or peripheral buses (2349) (e.g., a USB port on the computer device (2300); other systems are typically integrated into the core of the computer device (2300) via a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer device (2300) can communicate with other entities. The communication can be unidirectional, used only for receiving (e.g., wireless television), unidirectional, used only for sending (e.g., CAN bus to certain CAN bus devices), or bidirectional, such as through a local area or wide area digital network to other computer systems. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0206] The aforementioned human-computer interface device, human-accessible storage device, and network interface can be connected to the core (2340) of the computer device (2300).

[0207] The core (2340) may include one or more central processing units (CPU) (2341), graphics processing units (GPUs) (2342), dedicated programmable processing units in the form of field-programmable gate arrays (FPGAs) (2343), task-specific hardware accelerators (2344), etc. These devices, along with read-only memory (ROM) (2345), random access memory (2346), and internal mass storage (e.g., internal non-user-accessible hard disk drives, solid-state drives, etc.) (2347), can be connected via a system bus (2348). In some computer systems, the system bus (2348) can be accessed via one or more physical connectors to allow for expansion with additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (2348) or connected via a peripheral bus (2349). Peripheral bus architectures include external controller interfaces (PCI), universal serial buses (USB), etc. In one example, a screen (2310) may be connected to a graphics adapter (2350). Peripheral bus architectures include PCI, USB, etc.

[0208] The CPU (2341), GPU (2342), FPGA (2343), and accelerator (2344) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (2345) or RAM (2346). Transient data can also be stored in RAM (2346), while permanent data can be stored, for example, in internal mass storage (2347). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (2341), GPUs (2342), mass storage (2347), ROM (2345), RAM (2346), etc.

[0209] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be specifically designed and constructed for the purposes of this application, or they may be media and code well-known and usable by those skilled in the art of computer software.

[0210] By way of example and not limitation, a computer system having an architecture (2300), particularly a core (2340), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the non-volatile core (2340), such as internal mass storage (2347) or ROM (2345). Software implementing various embodiments of this application can be stored in such a device and executed by the core (2340). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can cause the core (2340), particularly the processor therein (including a CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described herein, including defining data structures stored in RAM (2346) and modifying such data structures according to a software-defined process. Alternatively or as an alternative, the computer system may provide logic hardwired or otherwise incorporated into circuitry (e.g., an accelerator (2344)) that may replace or operate with the software to perform the specific process or a specific portion of the specific process described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry storing the execution of software (such as an integrated circuit (IC)), circuitry containing execution logic, or both. This application includes any suitable combination of hardware and software.

[0211] Appendix: Acronyms

[0212] JEM: Joint Development Model

[0213] VVC: Next Generation Video Coding

[0214] BMS: Baseline Collection

[0215] MV: Motion Vector

[0216] HEVC: High-Efficiency Video Coding

[0217] SEI: Supplemental Enhancement Information

[0218] VUI: Video Availability Information

[0219] GOPs: Image Group

[0220] TUs: Transformation Unit

[0221] PUs: Prediction Units

[0222] CTUs: Coding Tree Units

[0223] CTBs: Coded Tree Blocks

[0224] PBs: Predicted Blocks

[0225] HRD: Assuming a reference decoder

[0226] SNR: Signal-to-noise ratio

[0227] CPUs: Central Processing Unit

[0228] GPUs: Graphics Processing Units

[0229] CRT: Cathode Ray Tube

[0230] LCD: Liquid Crystal Display

[0231] OLED: Organic Light Emitting Diode

[0232] CD: Optical Disc; DVD: Digital Video Disc

[0233] ROM: Read-Only Memory

[0234] RAM: Random Access Memory

[0235] ASIC: Application-Specific Integrated Circuit

[0236] PLD: Programmable Logic Device

[0237] LAN: Local Area Network

[0238] GSM: Global System for Mobile Communications

[0239] LTE: Long Term Evolution

[0240] CANBus: Controller Area Network Bus

[0241] USB: Universal Serial Bus

[0242] PCI: Peripheral Device Interconnect

[0243] FPGA: Field Programmable Gate Array

[0244] SSD: Field Programmable Gate Array

[0245] IC: Integrated Circuit

[0246] CU: Encoding Unit

[0247] While this application has described several exemplary embodiments, various modifications, arrangements, and equivalent substitutions of the embodiments are all within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are thus within its spirit and scope.

Claims

1. A method for processing a video by a neural network, characterized by, The method comprises: a processing circuit separates a first input channel that is piece-wise constant from other input channels of an input of a convolution operation, the first input channel comprising a quantization parameter, QP, map for a picture, and the other input channels comprising color components of the picture; the processing circuit computes a first intermediate output channel based on the other input channels of the input of the convolution operation by applying a reduced channel convolution operation on the color components of the picture using a kernel; and the processing circuit generates an output of the convolution operation based on a linear combination of the first intermediate output channel and the QP map for use in inter-picture prediction.

2. The method of claim 1, wherein, Further comprising: receiving the picture output from a deblocking filter.

3. The method of claim 1, wherein, Further comprising: applying an in-loop filter based on a dense residual convolutional neural network to the output of the convolution operation.

4. An apparatus for processing a video using a neural network, the apparatus comprising: The apparatus comprises a processing circuit configured to: separate a first input channel that is piece-wise constant from other input channels of an input of a convolution operation, the first input channel comprising a quantization parameter, QP, map for a picture, and the other input channels comprising color components of the picture; compute a first intermediate output channel based on the other input channels of the input of the convolution operation by applying a reduced channel convolution operation on the color components of the picture using a kernel; and generate an output of the convolution operation based on a linear combination of the first intermediate output channel and the QP map for use in inter-picture prediction. The processing circuit is further configured to:

5. The apparatus of claim 4, wherein, receive the picture output from a deblocking filter. The processing circuit is further configured to:

6. The apparatus of claim 4, wherein, apply an in-loop filter based on a dense residual convolutional neural network to the output of the convolution operation. The instructions, when executed by a computer, cause the computer to perform the method of any one of claims 1-3.

7. A non-transitory computer readable medium storing instructions, wherein the instructions, when executed by a processor, cause the processor to perform operations comprising: Comprising:

8. A computer device, comprising: a processor and a memory; the memory stores computer code which, when executed by the processor, causes the processor to perform the method of any one of claims 1-3. ​

Images