Video coding method, computer readable medium and electronic device

Abstract

Embodiments of the present application provide a video coding method, device, computer readable medium and electronic device. The video decoding method comprises: decoding a video bitstream to determine whether a to-be-filtered image is subjected to collaborative optimization processing using a neural network based loop filter (NNLF) and an adaptive loop filter (ALF); if the to-be-filtered image is subjected to collaborative optimization processing using the NNLF and the ALF, skipping a filtering process related to a filtering result of the NNLF in the ALF, and using the filtering result of the NNLF as a processing result of the related filtering process in the ALF. The technical solution of the embodiments of the present application can improve the filtering effect through collaborative optimization processing of the NNLF and the ALF, and can use the filtering result of the NNLF as the processing result of the related filtering process in the ALF, thereby reducing unnecessary calculation amount and improving the overall coding efficiency.

Description

Technical Field

[0001] This application relates to the fields of computer and communication technology, and more specifically, to a video encoding / decoding method, apparatus, computer-readable medium, and electronic device. Background Technology

[0002] In the field of video encoding and decoding, after the predicted image and the reconstructed residual image are superimposed to generate the reconstructed image, the reconstructed image will be distorted. Therefore, in order to obtain a higher quality image, it is usually necessary to perform loop filtering on the reconstructed image. In the loop filtering process, how to improve the filtering effect and improve the encoding and decoding efficiency at the same time is an urgent technical problem to be solved. Summary of the Invention

[0003] The embodiments of this application provide a video encoding / decoding method, apparatus, computer-readable medium, and electronic device, which can improve the filtering effect through the collaborative optimization processing of NNLF and ALF, and can use the filtering result of NNLF as the processing result of the related filtering process in ALF, reducing unnecessary computation and improving the overall encoding / decoding efficiency.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of this application.

[0005] In a first aspect, embodiments of this application provide a video decoding method, comprising: decoding a video bitstream to determine whether an image to be filtered is subjected to collaborative optimization processing using a neural network-based loop filter (NNLF) and an adaptive loop filter (ALF); if the image to be filtered is subjected to collaborative optimization processing using the NNLF and the ALF, then skipping the filtering process in the ALF related to the filtering result of the NNLF, and using the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

[0006] Secondly, embodiments of this application provide a video coding method, comprising: determining whether an image to be filtered uses NNLF and ALF for collaborative optimization processing based on a rate-distortion optimization criterion; if it is determined that the image to be filtered uses NNLF and ALF for collaborative optimization processing, encoding a flag bit to indicate that the image to be filtered uses NNLF; skipping the filtering process in ALF related to the filtering result of NNLF, and using the filtering result of NNLF as the processing result of the related filtering process in ALF.

[0007] Thirdly, embodiments of this application provide a video decoding apparatus, including: a decoding unit configured to decode a video bitstream to determine whether an image to be filtered uses a neural network-based loop filter (NNLF) and an adaptive loop filter (ALF) for collaborative optimization processing; and a processing unit configured to, if the image to be filtered uses the NNLF and the ALF for collaborative optimization processing, skip the filtering process in the ALF related to the filtering result of the NNLF, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

[0008] Fourthly, embodiments of this application provide a video encoding apparatus, comprising: a determining unit configured to determine, using a pass-through distortion optimization criterion, whether an image to be filtered uses NNLF and ALF for collaborative optimization processing; an encoding unit configured to, if it is determined that the image to be filtered uses NNLF and ALF for collaborative optimization processing, encode a flag bit to indicate that the image to be filtered uses NNLF; and a processing unit configured to skip the filtering process in ALF related to the filtering result of NNLF, and use the filtering result of NNLF as the processing result of the related filtering process in ALF.

[0009] Fifthly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the video decoding method or video encoding method as described in the above embodiments.

[0010] Sixthly, embodiments of this application provide an electronic device, including: one or more processors; and a storage device for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the video decoding method or video encoding method as described in the above embodiments.

[0011] In a seventh aspect, embodiments of this application provide a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of an electronic device reads from and executes the computer program from the computer-readable storage medium, causing the electronic device to perform the video decoding or video encoding methods provided in the various alternative embodiments described above.

[0012] Eighthly, embodiments of this application provide a method for storing a video stream, wherein the video stream is decoded according to the video decoding method described in the above embodiments, or the video stream is generated according to the video encoding method described in the above embodiments.

[0013] In some embodiments of this application, the video decoding end can decode the video stream to determine whether the image to be filtered uses NNLF and ALF for collaborative optimization. If the image to be filtered uses NNLF and ALF for collaborative optimization, the filtering process related to the NNLF filtering result in ALF is skipped, and the NNLF filtering result is used as the processing result of the related filtering process in ALF. Therefore, the technical solution of this application, by introducing NNLF and ALF for collaborative optimization, can leverage the powerful capabilities of NNLF combined with ALF to effectively improve the quality of the reconstructed image. By skipping the filtering process related to the NNLF filtering result in ALF and using the NNLF filtering result as the processing result of the related filtering process in ALF, repeated filtering processes in ALF are avoided, unnecessary computation is reduced, and the overall efficiency of the encoding and decoding process is improved.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0015] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0016] Figure 2 This diagram illustrates the placement of the video encoding and decoding devices in a streaming system.

[0017] Figure 3 A basic flowchart of a video encoder is shown;

[0018] Figure 4 A schematic diagram of filtering based on NNLF is shown;

[0019] Figure 5 A schematic diagram of filtering based on NNLF is shown;

[0020] Figure 6 A schematic diagram of filtering the luminance and chrominance components is shown.

[0021] Figure 7 A flowchart of a video decoding method according to an embodiment of this application is shown;

[0022] Figures 8 to 13 A structural diagram of a loop filter according to an embodiment of this application is shown;

[0023] Figure 14A flowchart of a video encoding method according to an embodiment of this application is shown;

[0024] Figure 15 A block diagram of a video decoding apparatus according to an embodiment of this application is shown;

[0025] Figure 16 A block diagram of a video encoding apparatus according to an embodiment of this application is shown;

[0026] Figure 17 A schematic diagram of the structure of a computer system suitable for implementing the computer device of the present application is shown. Detailed Implementation

[0027] Exemplary embodiments will now be described in a more comprehensive manner with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to these examples; rather, these embodiments are provided so that this application will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art.

[0028] Furthermore, the features, structures, or characteristics described in this application can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to provide a full understanding of the embodiments of this application. However, those skilled in the art will recognize that when implementing the technical solutions of this application, not all the detailed features in the embodiments may be used, one or more specific details may be omitted, or other methods, elements, devices, steps, etc., may be employed.

[0029] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0032] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0034] like Figure 1 As shown, system architecture 100 includes multiple terminal devices that can communicate with each other via, for example, a network 150. For instance, system architecture 100 may include a first terminal device 110 and a second terminal device 120 interconnected via network 150. Figure 1 In one embodiment, the first terminal device 110 and the second terminal device 120 perform unidirectional data transmission.

[0035] For example, the first terminal device 110 can encode video data (e.g., a video image stream captured by the terminal device 110) to transmit it to the second terminal device 120 via the network 150. The encoded video data is transmitted in the form of one or more encoded video streams. The second terminal device 120 can receive the encoded video data from the network 150, decode the encoded video data to recover the video data, and display video images based on the recovered video data.

[0036] In one embodiment of this application, system architecture 100 may include a third terminal device 130 and a fourth terminal device 140 that perform bidirectional transmission of encoded video data, such as during a video conference. For bidirectional data transmission, each of the third terminal device 130 and the fourth terminal device 140 may encode video data (e.g., a video image stream captured by the terminal device) for transmission over network 150 to the other terminal device. Each of the third terminal device 130 and the fourth terminal device 140 may also receive encoded video data transmitted by the other terminal device, decode the encoded video data to recover the video data, and display the video images on an accessible display device based on the recovered video data.

[0037] exist Figure 1 In the embodiments shown, the first terminal device 110, the second terminal device 120, the third terminal device 130 and the fourth terminal device 140 may be servers or terminals, but the principles disclosed in this application are not limited to these.

[0038] Servers can be standalone physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals can be smartphones, tablets, laptops, desktop computers, smart speakers, smart voice interaction devices, smartwatches, smart home appliances, in-vehicle terminals, aircraft, etc., but are not limited to these.

[0039] Figure 1 The network 150 shown represents any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between the first terminal device 110, the second terminal device 120, the third terminal device 130, and the fourth terminal device 140. The communication network 150 may exchange data in circuit-switched and / or packet-switched channels. This network may include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this application, unless explained below, the architecture and topology of network 150 may be irrelevant to the operation of the disclosure herein.

[0040] In one embodiment of this application, Figure 2 The illustration shows the placement of video encoding and decoding devices in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital television (TV), and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0041] The streaming system may include an acquisition subsystem 213, which may include a video source 201 such as a digital camera, which creates an uncompressed video image stream 202. In an embodiment, the video image stream 202 includes samples captured by a digital camera. The video image stream 202 is depicted as a thick line to emphasize the high data volume of the video image stream compared to encoded video data 204 (or encoded video bitstream 204). The video image stream 202 may be processed by an electronic device 220, which includes a video encoding device 203 coupled to the video source 201. The video encoding device 203 may include hardware, software, or a combination of hardware and software to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video data 204 (or encoded video bitstream 204) is depicted as a thin line to emphasize the lower data volume of the encoded video data 204 (or encoded video bitstream 204), which may be stored on a streaming server 205 for future use. One or more streaming client subsystems, such as Figure 2 Client subsystems 206 and 208 can access streaming server 205 to retrieve copies 207 and 209 of encoded video data 204. Client subsystem 206 may include, for example, a video decoding device 210 in electronic device 230. Video decoding device 210 decodes the incoming copy 207 of the encoded video data and produces an output video picture stream 211 that can be displayed on display 212 (e.g., a screen) or another presentation device. In some streaming systems, the encoded video data 204, video data 207, and video data 209 (e.g., video stream) may be encoded according to certain video encoding / compression standards.

[0042] It should be noted that electronic devices 220 and 230 may include other components not shown in the figures. For example, electronic device 220 may include a video decoding device, and electronic device 230 may also include a video encoding device.

[0043] In one embodiment of this application, taking High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) from international video coding standards, as well as the Chinese national video coding standard AVS, as examples, after an input video frame image, the video frame image is divided into several non-overlapping processing units according to a block size. Each processing unit performs a similar compression operation. This processing unit is called a Coding Tree Unit (CTU) or Largest Coding Unit (LCU). The CTU can be further subdivided into more refined units to obtain one or more basic Coding Units (CUs). The CU is the most basic element in a coding process.

[0044] In another embodiment, this processing unit can also be called a tile, which is a rectangular area of ​​a multimedia data frame that can be independently decoded and encoded. In the Alliance for Open Media Video 1 (AV1) standard, the tile can be further subdivided into one or more superblocks (SBs). The SB is the starting point for block partitioning and can be further divided into multiple subblocks. The superblocks are then further subdivided into one or more blocks. Each block is the most basic element in a coding process. Optionally, an SB can contain several blocks (Bs).

[0045] The above method of dividing video frame images can be called a block partition structure. The following introduces some concepts in the encoding process:

[0046] Predictive coding includes intra-frame prediction and inter-frame prediction. The original video signal is predicted from a selected reconstructed video signal to obtain a residual video signal. The encoder needs to decide which predictive coding mode to choose for the current coding unit (or coding block) and inform the decoder. Intra-frame prediction refers to the predicted signal coming from a region within the same image that has already been encoded and reconstructed; inter-frame prediction refers to the predicted signal coming from another encoded image (called a reference image) that is different from the current image.

[0047] Transform and Quantization: After the residual video signal undergoes transformation operations such as Discrete Fourier Transform (DFT) and Discrete Cosine Transform (DCT), the signal is transformed into the transform domain, and these are called transform coefficients. The transform coefficients are then subjected to lossy quantization, losing some information to make the quantized signal more suitable for compression. In some video coding standards, there may be more than one transform method to choose from. Therefore, the encoder needs to select one of the transform methods for the current coding unit (or coding block) and inform the decoder. The fineness of quantization is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a wider range of values ​​will be quantized into the same output, which usually leads to greater distortion and a lower bit rate. Conversely, a smaller QP value means that coefficients with a smaller range of values ​​will be quantized into the same output, which usually leads to less distortion and a higher bit rate.

[0048] Entropy coding, or statistical coding, involves statistically compressing the quantized transform-domain signal based on the frequency of each value, ultimately outputting a binary (0 or 1) compressed bitstream. Simultaneously, other information generated during encoding, such as the selected coding mode and motion vector data, also requires entropy coding to reduce the bit rate. Statistical coding is a lossless coding method that effectively reduces the bit rate required to represent the same signal. Common statistical coding methods include Variable Length Coding (VLC) and Content-Adaptive Binary Arithmetic Coding (CABAC).

[0049] Context-Based Binary Arithmetic Coding (CABAC) primarily involves three steps: binarization, context modeling, and binary arithmetic coding. After binarizing the input syntax elements, the binary data can be encoded using either a regular coding mode or a bypass coding mode. The bypass coding mode eliminates the need to assign a specific probability model to each binary bit; the input binary bit bin value is directly encoded using a simple bypass encoder, thus accelerating the overall encoding and decoding speed. Generally, different syntax elements are not completely independent, and even identical syntax elements possess a certain degree of memory. Therefore, according to conditional entropy theory, using other encoded syntax elements for conditional coding can further improve coding performance compared to independent coding or memoryless coding. This encoded symbol information used as conditions is called the context. In the regular coding mode, the binary bits of the syntax elements sequentially enter the context modeler. The encoder assigns an appropriate probability model to each input binary bit based on the values ​​of previously encoded syntax elements or binary bits; this process is called context modeling. The context model corresponding to a grammatical element can be located using the context index increment (ctxIdxInc) and the context index start (ctxIdxStart). After the bin value and the assigned probability model are fed into the binary arithmetic encoder for encoding, the context model needs to be updated based on the bin value, which is the adaptive process in encoding.

[0050] Loop Filtering: The transformed and quantized signal undergoes inverse quantization, inverse transform, and prediction compensation to obtain a reconstructed image. Due to the effects of quantization, the reconstructed image differs from the original image in some aspects, resulting in distortion. Therefore, filtering operations such as deblocking filters (DBF), sample adaptive offset (SAO), or adaptive loop filters (ALF) can effectively reduce the distortion caused by quantization. Since these filtered reconstructed images serve as a reference for subsequent coded images in predicting future image signals, the aforementioned filtering operations are also called loop filtering, i.e., filtering operations within the coding loop.

[0051] In one embodiment of this application, Figure 3 A basic flowchart of a video encoder is shown, illustrating the process using intra-frame prediction as an example. The original image signal s...k [x,y] and the predicted image signal Perform the difference operation to obtain the residual signal u. k [x,y], residual signal u k After transformation and quantization, [x,y] is obtained as quantization coefficients. These coefficients are then used to obtain the encoded bitstream through entropy encoding, and to obtain the reconstructed residual signal u' through inverse quantization and inverse transform. k [x,y], predict image signal With the reconstructed residual signal u' k [x,y] superimposed to generate image signals Image signal On one hand, the signal is input to the intra-frame mode decision module and the intra-frame prediction module for intra-frame prediction processing; on the other hand, the reconstructed image signal s' is output through loop filtering. k [x,y], reconstruct the image signal s' k [x,y] can be used as a reference image for the next frame for motion estimation and motion compensation prediction. Then, based on the result s' of the motion compensation prediction... r [x+m x ,y+m y ] and intra-frame prediction results Obtain the predicted image signal for the next frame. And continue repeating the above process until the coding is complete.

[0052] Based on the above encoding process, at the decoding end, for each coding unit (or coding block), after acquiring the compressed bitstream (i.e., bitstream), entropy decoding is performed to obtain various mode information and quantization coefficients. Then, the quantization coefficients undergo inverse quantization and inverse transform processing to obtain the residual signal. On the other hand, based on the known coding mode information, the prediction signal corresponding to the coding unit (or coding block) can be obtained. Then, the residual signal and the prediction signal are added together to obtain the reconstructed signal. The reconstructed signal then undergoes loop filtering and other operations to generate the final output signal. Furthermore, in this series of encoding processes, the coding framework mainly relies on the Rate Distortion Optimization (RDO) criterion to evaluate and select the optimal coding parameters and results.

[0053] The aforementioned loop filtering process can be implemented using a neural network loop filter (NNLF). For example... Figure 4 As shown, after the NNLF is trained, the image to be filtered can be input into the trained NNLF to obtain the filtered image. Furthermore, to accelerate training and enhance generalization ability, residual learning can also be used to train the NNLF, such as... Figure 5As shown, the image to be filtered is input into NNLF to learn the filtered residual image, and then the image to be filtered and the filtered residual image are added together to obtain the filtered image.

[0054] NNLF typically uses a loss function to constrain the filtered image, aiming to restore it as closely as possible to the original image. The loss function measures the difference between the predicted and true values; a larger loss value indicates a greater difference, and the training objective is to reduce this loss value. Optionally, NNLF can use L1-norm and L2-norm loss functions during training.

[0055] To flexibly separate the luminance and chrominance components of an image, NNLF can also use the Separation of Luma and Chroma (SLC) method. For example, ... Figure 6 As shown, the luminance component (Y) and chrominance components (Cb and Cr) are constructed using different network structures (or the same network structure can be used). During training, the luminance component (Y) and chrominance components (Cb and Cr) can be trained separately to improve the filtering performance for the luminance component and chrominance component.

[0056] The ALF mentioned in the above embodiments is a Wiener filter. Based on the Wiener filtering principle, it constructs the Wiener-Hough equation using the original image information and the reconstructed image information to be filtered, and solves for a series of filter coefficients with the minimum mean square error, thereby reducing decoding errors and improving image quality. Specifically, ALF can derive optimal coefficients based on a filter template of a certain shape, utilizing the correlation between the coefficients and surrounding pixels within the coverage area of ​​the filter template. This improves the quality of pixels after ALF filtering, reduces the difference from the original pixels, and thus enhances the image.

[0057] The input to ALF is typically the reconstructed image after SAO processing. However, in the latest international video coding standard—Enhanced Compression Model (ECM)—the reconstructed image before DBF and the residual image can also be used as input to ALF to further improve coding performance. In other words, ALF not only processes the SAO-processed image using a filter template of a certain shape, but also uses filter templates of other shapes to process the reconstructed image and the residual image before DBF. The residual image is the difference between the predicted image and the original image during video coding, i.e., the true residual parsed at the decoding end.

[0058] In recent years, an increasing number of neural network filtering tools have been integrated into traditional codec platforms to further improve video encoding performance. However, there is still room for improvement in the synergistic optimization of neural network filtering tools (such as NNLF) and traditional filtering tools. Based on this, the technical solution of this application proposes synergistic optimization processing using NNLF and ALF, thereby leveraging the powerful capabilities of NNLF in combination with ALF to effectively improve the quality of reconstructed images.

[0059] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0060] Figure 7 A flowchart of a video decoding method according to an embodiment of this application is shown. This video decoding method can be executed by a device with computing processing capabilities, such as a terminal device or a server. (Refer to...) Figure 7 As shown, this video decoding method includes at least S710 to S720, which are described in detail below:

[0061] In the S710, the video stream is decoded to determine whether the image to be filtered should be processed using NNLF and ALF for collaborative optimization.

[0062] It should be noted that the image to be filtered can be a frame-level image or a block-level image. In other words, NNLF and ALF can be used for collaborative optimization of frame-level images (such as the current frame image to be filtered), or NNLF and ALF can be used for collaborative optimization of block-level images (such as the current block image to be filtered).

[0063] Optionally, the image to be filtered can be an image of one component or multiple components, such as an image of the luminance component Y, an image of the chrominance component U, or an image of the chrominance component V, etc.

[0064] In some optional embodiments, the process of decoding the video stream to determine whether the image to be filtered uses NNLF and ALF for collaborative optimization can specifically be as follows: A first flag bit and a second flag bit are decoded from the video stream. The first flag bit indicates whether NNLF is enabled, and the second flag bit indicates whether ALF is enabled. If the first flag bit indicates that NNLF is enabled and the second flag bit indicates that ALF is enabled, a third flag bit can be decoded from the video stream. This third flag bit indicates whether the image to be filtered uses NNLF. If the third flag bit indicates that the image to be filtered uses NNLF, then it is determined that the image to be filtered uses NNLF and ALF for collaborative optimization.

[0065] Optionally, if the value of the first flag bit is 1, it indicates that NNLF is enabled; if the value of the first flag bit is 0, it indicates that NNLF is not enabled. Similarly, if the value of the second flag bit is 1, it indicates that ALF is enabled; if the value of the second flag bit is 0, it indicates that ALF is not enabled. Optionally, the first and second flag bits can be sequence-level flag bits, i.e., used to indicate whether NNLF and ALF are enabled in the current sequence; or the first and second flag bits can also be frame-level flag bits, i.e. used to indicate whether NNLF and ALF are enabled in the current frame; or the first and second flag bits can also be stripe-level flag bits, i.e. used to indicate whether NNLF and ALF are enabled in the current stripe, etc.

[0066] Optionally, if the third flag is a first value (e.g., 0), it can indicate that the image to be filtered does not use NNLF; if the third flag is another value (e.g., a non-zero value), it indicates that the image to be filtered uses NNLF and indicates the quantization parameter offset corresponding to the image to be filtered.

[0067] It should be noted that the quantization parameter offset (QPOffset) in video coding is mainly used to adjust the fineness of the quantization process, thereby affecting video compression efficiency and the quality of the reconstructed image. The quantization parameter (QP) determines the fineness of the quantization process. A smaller QP value means finer quantization, resulting in lower distortion and higher image quality, but also a higher bitrate; a larger QP value, on the other hand, leads to higher distortion and lower image quality, but can achieve a higher compression ratio and a lower bitrate. In the embodiments of this application, the quantization parameter offset is an adjustment to the basic quantization parameter when using NNLF. Specifically, by changing the QP value, the degree of information loss during the encoding process can be controlled to a certain extent to achieve better rate-distortion performance. For example, if NNLF can significantly improve a certain type of distortion, the quantization distortion can be reduced by decreasing the QP value (i.e., a negative QPOffset, such as QPOffset = -5); conversely, if a further reduction in bitrate is desired, the QP value can be increased (i.e., a positive QPOffset, such as QPOffset = +5).

[0068] In some optional embodiments, if the third flag is a second value (e.g., 1), it can indicate that the image to be filtered uses NNLF and that the quantization parameter offset corresponding to the image to be filtered is a first offset value. Optionally, the first offset value can be, for example, 0, or other integer values.

[0069] In some optional embodiments, if the third flag is a third value (e.g., 2), it can indicate that the image to be filtered uses NNLF and that the quantization parameter offset corresponding to the image to be filtered is the second offset value. Optionally, the second offset value can be, for example, -5 / +5, or other integer values.

[0070] Optionally, the second offset value for lower time domain layers can be negative, such as -5; the second offset value for higher time domain layers can be positive, such as +5.

[0071] It's important to note that the low temporal layer typically refers to the base layer or a lower temporal resolution layer. The low temporal layer contains essential information sufficient to provide a basic video playback experience. For a given video sequence, the low temporal layer often has a lower frame rate, meaning it provides slower playback speed, but it can guarantee a basic visual experience even with limited bandwidth. Setting the offset value to negative for the low temporal layer is primarily to use finer quantization (i.e., a smaller QP value) to reduce distortion. This helps improve the quality of the low temporal layer, as this part is often the foundation for ensuring a basic viewing experience.

[0072] Higher temporal layers add more detail and higher temporal resolution on top of lower temporal layers. This extra information makes video playback smoother and provides a better viewing experience. For higher temporal layers, setting the offset value to a positive value means allowing a larger quantization step size (i.e., a larger QP value) in exchange for higher compression efficiency. This is because higher temporal layers are mainly used to enhance the viewing experience rather than essential basic playback functions, so they can tolerate higher distortion to achieve a higher compression ratio, especially when bandwidth is limited.

[0073] In some optional embodiments, if the third flag is a fourth value (e.g., 3), it can indicate that some or all blocks of the image to be filtered use NNLF, and the quantization parameter offset corresponding to the image to be filtered is the third offset value. Optionally, the third offset value can be, for example, 0 / -5 / +5, or other integer values.

[0074] In other embodiments of this application, the third flag bit may also be used to indicate the quantization parameter offset used when using NNLF by other values.

[0075] In S720, if the image to be filtered is processed by NNLF and ALF in a coordinated optimization process, the filtering process in ALF related to the filtering result of NNLF is skipped, and the filtering result of NNLF is used as the processing result of the related filtering process in ALF.

[0076] It should be noted that since ALF performs coefficient training and filtering processes based on the reconstructed image, coefficient training and filtering processes based on the residual image, and coefficient training and filtering processes based on the SAO output, if the NNLF filtering result is the filtered reconstructed image, then the coefficient training and filtering processes based on the reconstructed image in ALF can be skipped, and the NNLF-filtered reconstructed image can be used to replace the corresponding processing result in ALF. Similarly, if the NNLF filtering result is the filtered residual image, then the coefficient training and filtering processes based on the residual image in ALF can be skipped, and the NNLF-filtered residual image can be used to replace the corresponding processing result in ALF. Based on this, the implementation process of S720 can have the following embodiments:

[0077] In some optional embodiments, the coefficient training and filtering process based on the reconstructed image in ALF can be skipped, and the reconstructed image filtered by NNLF can be used as the processing result of the coefficient training and filtering process based on the reconstructed image in ALF. The technical solution of this embodiment is to use the reconstructed image filtered by NNLF to replace the processing result of the coefficient training and filtering process based on the reconstructed image in ALF.

[0078] In some optional embodiments, the coefficient training and filtering process based on the residual image in ALF can be skipped, and the residual image filtered by NNLF can be used as the processing result of the coefficient training and filtering process based on the residual image in ALF. The technical solution of this embodiment is to use the residual image filtered by NNLF to replace the processing result of the coefficient training and filtering process based on the reconstructed image in ALF.

[0079] In some optional embodiments, the coefficient training and filtering process based on the reconstructed image and residual image in ALF can be skipped, and the reconstructed image and residual image filtered by NNLF can be used as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in ALF. The technical solution of this embodiment is to use the reconstructed image filtered by NNLF to replace the processing result of the coefficient training and filtering process based on the reconstructed image in ALF, and to use the residual image filtered by NNLF to replace the processing result of the coefficient training and filtering process based on the residual image in ALF.

[0080] In some optional embodiments, the coefficient training and filtering process based on the residual image in ALF can be skipped, and the residual image obtained based on the predicted image and the reconstructed image after NNLF filtering can be used as the processing result of the coefficient training and filtering process based on the residual image in ALF. The main technical solution of this embodiment is that after obtaining the filtered reconstructed image through NNLF, a residual image is calculated using the predicted image and the filtered reconstructed image, and then this residual image is used to replace the processing result of the coefficient training and filtering process based on the residual image in ALF.

[0081] Optionally, in the above embodiments, the reconstructed image mentioned in the process of skipping coefficient training and filtering based on the reconstructed image in ALF can be the reconstructed image before DBF, that is, the reconstructed image without DBF filtering. In other embodiments of this application, it can also be the reconstructed image after DBF filtering, or it can also be the reconstructed image after SAO filtering.

[0082] Each of the above embodiments can be selected as the default processing method. Alternatively, in one embodiment of this application, a fourth flag bit can be decoded from the video bitstream. This fourth flag bit is used to indicate the cooperative optimization method of NNLF and ALF, which represents the skipped filtering processes in ALF. In other words, the value of the fourth flag bit can indicate the specific skipped filtering processes in ALF.

[0083] Optionally, if the fourth flag is the first value (e.g., 0), the coefficient training and filtering process based on the reconstructed image in ALF can be skipped, and the reconstructed image after NNLF filtering can be used as the processing result of the coefficient training and filtering process based on the reconstructed image in ALF.

[0084] Optionally, if the fourth flag is the second value (e.g., 1), the coefficient training and filtering process based on the residual image in ALF can be skipped, and the residual image after NNLF filtering can be used as the processing result of the coefficient training and filtering process based on the residual image in ALF.

[0085] Optionally, if the fourth flag is the third value (e.g., 2), the coefficient training and filtering process based on the reconstructed image and residual image in ALF is skipped, and the reconstructed image and residual image after NNLF filtering are used as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in ALF.

[0086] It should be noted that the technical solutions of the above embodiments of this application can be applied to various loop filter structures combining NNLF and ALF, for example, they can be applied to Figures 8 to 13 The structure of the loop filter shown in any of the figures.

[0087] Specifically, such as Figure 8 As shown, the input of the NNLF is connected to the input of the DBF to receive the image to be filtered (such as the reconstructed image to be filtered). The output of the DBF is connected to the input of the SAO, the output of the SAO is connected to the first input of the ALF, the output of the NNLF is connected to the second input of the ALF, and the output of the ALF is used to output the filtered image. In this example, if it is necessary to skip the coefficient training and filtering process based on the reconstructed image in the ALF, then the coefficient training and filtering process based on the reconstructed image before the DBF in the ALF can be skipped.

[0088] like Figure 9 As shown, the input of the NNLF is connected to the input of the DBF to receive the image to be filtered (such as the reconstructed image to be filtered). The output data of the DBF and the output data of the NNLF are fused together and used as the input data of the SAO. The output of the SAO is connected to the first input of the ALF, the output of the NNLF is connected to the second input of the ALF, and the output of the ALF is used to output the filtered image. Figure 9 In the illustrated embodiment, NNLF can not only be co-optimized with ALF, but also combined with DBF. Optionally, the output data of DBF and the output data of NNLF can be weighted and fused according to a set weight.

[0089] For example, the output data of DBF and NNLF can be fused using the following formula: R refine =w×R NNLF +(1-w)×R DBF Among them, R NNLF R represents the image after NNLF filtering (i.e., the output data of NNLF); DBF represents the image after DBF filtering (i.e., the output data of DBF); w represents the weighting factor, i.e., R. NNLF The weights of R; refine This represents the fused result, which is the reconstructed image input into SAO.

[0090] like Figure 10 As shown, the input of the DBF is used to receive the image to be filtered (such as the reconstructed image to be filtered). The output of the DBF is connected to the input of the SAO. The output of the SAO is connected to the first input of the ALF and the input of the NNLF. The output of the NNLF is connected to the second input of the ALF. The output data of the ALF and the output data of the NNLF are fused together to form the filtered image. In this embodiment, the input of the NNLF is the reconstructed image after SAO filtering. Optionally, the output data of the ALF and the output data of the NNLF can be weighted and fused according to a set weight.

[0091] exist Figure 10 In one modified embodiment, such as Figure 11 As shown, the ALF output data can be used as the filtered image.

[0092] like Figure 12 As shown, the input of the NNLF is connected to the input of the DBF to receive the image to be filtered (such as the reconstructed image to be filtered); the output of the DBF is connected to the input of the SAO, and the output data of the SAO and the output data of the NNLF are fused together to serve as the input data for the first input of the ALF; the output of the NNLF is connected to the second input of the ALF, and the output of the ALF is used to output the filtered image. Optionally, the output data of the SAO and the output data of the NNLF can be weighted and fused according to a set weight.

[0093] like Figure 13 As shown, the input of the DBF is used to receive the image to be filtered (such as the reconstructed image to be filtered). The output of the DBF is connected to the input of the SAO and the input of the NNLF. The output data of the SAO and the output data of the NNLF are fused together and used as the input data of the first input of the ALF. The output of the NNLF is connected to the second input of the ALF, and the output of the ALF is used to output the filtered image. In this embodiment, the input of the NNLF is the reconstructed image filtered by the DBF. Optionally, the output data of the SAO and the output data of the NNLF can be weighted and fused according to a set weight.

[0094] Figure 7 This description focuses on the technical solutions of the embodiments of this application from the perspective of video decoding. The following is a combination of... Figure 14 The technical solutions of the embodiments of this application will be described again from the perspective of video encoding.

[0095] Figure 14 A flowchart of a video encoding method according to an embodiment of this application is shown. This video encoding method can be executed by a device with computing processing capabilities, such as a terminal device or a server. (Refer to...) Figure 14 As shown, this video encoding method includes at least S1410 to S1430, which are described in detail below:

[0096] In S1410, the rate-distortion optimization criterion is used to determine whether the image to be filtered should be processed using NNLF and ALF for collaborative optimization.

[0097] In S1420, if it is determined that the image to be filtered uses NNLF and ALF for collaborative optimization processing, a flag bit is encoded to indicate that the image to be filtered uses NNLF.

[0098] In S1430, the filtering process related to the filtering result of NNLF in ALF is skipped, and the filtering result of NNLF is used as the processing result of the related filtering process in ALF.

[0099] It should be noted that the processing at the video encoding end is similar to that at the video decoding end. For details, please refer to the aforementioned processing at the decoding end, which will not be repeated here.

[0100] The technical solution of this application mainly introduces NNLF and ALF for collaborative optimization processing, so as to make full use of the effective filtering information generated by NNLF and thereby improve video coding performance.

[0101] The following is Figure 8 Taking the loop filter structure shown as an example, the implementation details of the technical solution of this application embodiment will be described in detail:

[0102] Reference Figure 8 As shown, the input to NNLF is the reconstructed image before DBF, and the output is the image after NNLF filtering. In some optional embodiments, the decoder can parse the sequence-level flags sps_nnlf_enabled_flag and sps_alf_enabled_flag to obtain the enabling status of NNLF and ALF for the entire sequence, respectively. Specific values ​​and meanings are shown in Table 1.

[0103] Flag meaning sps_nnlf_enabled_flag 0 indicates NNLF is disabled, 1 indicates NNLF is enabled. sps_alf_enabled_flag 0 indicates ALF is disabled, 1 indicates NNLF is enabled.

[0104] Table 1

[0105] Referring to Table 1, when sps_nnlf_enabled_flag = 0, NNLF processing is skipped; when sps_nnlf_enabled_flag = 1, NNLF processing is enabled. If sps_alf_enabled_flag = 1, meaning ALF processing is also enabled, then the frame-level NNLF flag slice_nnlf_unified_mode can be parsed to confirm the NNLF usage status of the current frame.

[0106] Optionally, the specific values ​​and meanings of slice_nnlf_unified_mode are shown in Table 2:

[0107]

[0108] Table 2

[0109] Referring to Table 2, when slice_nnlf_unified_mode = 0, NNLF does not perform co-optimization with ALF, and the decoder performs loop filtering according to the DBF->SAO->ALF flow. When slice_nnlf_unified_mode is not equal to 0, i.e., slice_nnlf_unified_mode is 1, 2, or 3, NNLF can perform co-optimization with ALF according to the default method or the flag bit transmission method.

[0110] In some optional embodiments, the process of NNLF co-optimizing with ALF according to the default method can be any of the following:

[0111] Skip the coefficient training and filtering process in ALF based on the image before DBF (i.e., the reconstructed image without DBF filtering), and use the NNLF filtered image as the output of this step (i.e., the coefficient training and filtering process in ALF based on the image before DBF).

[0112] Skip the residual-based coefficient training and filtering process in ALF, and use the residual after NNLF filtering as the output of this step (i.e., the residual-based coefficient training and filtering process in ALF).

[0113] Skip the residual-based coefficient training and filtering in ALF, and use the difference between the predicted image and the reconstructed image after NNLF filtering as the output of this step (i.e., the residual-based coefficient training and filtering in ALF).

[0114] Skip the coefficient training and filtering processing of the ALF based on the pre-DBF image and the residual image, and use the output results corresponding to the NNLF filtered image and the filtered residual by default.

[0115] In some optional embodiments, the process of NNLF co-optimizing with ALF based on the flag bit transmission method can be as follows: parsing the frame-level flag bit slice_nnlf_plus_alf_mode to determine the co-optimization working mode of NNLF and ALF based on the value of slice_nnlf_plus_alf_mode. The specific values ​​and meanings of slice_nnlf_plus_alf_mode are shown in Table 3.

[0116]

[0117] Table 3

[0118] In some alternative embodiments, for the encoding end, if it enters the loop filtering stage, then as Figure 8As shown, the reconstructed image before DBF is input into DBF and NNLF for processing, respectively. During NNLF processing, the Rate-Distortion Optimization (RDO) criterion is used to calculate the RD cost for different methods: no NNLF application, full-frame NNLF application with QPOffset=0, full-frame NNLF application with QPOffset=+5 / -5, and block-level NNLF application with QPOffset=0 / +5 / -5. The method with the lowest RD cost is selected as the final application method for NNLF, and the frame-level flag slice_nnlf_unified_mode is transmitted to the decoding end. When slice_nnlf_unified_mode is set to 0, no additional processing is performed; conversely, if slice_nnlf_unified_mode is not equal to 0, one of the three methods described in the decoding end can be used for co-optimization processing with ALF.

[0119] It should be noted that the above embodiments are based on Figure 8 The description uses the loop filter structure shown as an example. In other embodiments of this application, for example, Figures 9 to 13 The loop filter structure shown can also achieve synergistic optimization processing of NNLF and ALF, except that NNLF can also be combined with DBF or SAO. Furthermore, the technical solutions of the above embodiments can be used for filtering at least one of the luminance component Y and chrominance components U and V. The sequence-level and frame-level flag bits mentioned in the above embodiments are only specific examples; in other embodiments of this application, at least one of sequence-level, frame-level, strip-level, block-level, CTU-level, CU-level, PU-level, and TU-level flag bits can also be used.

[0120] The technical solutions of the above embodiments of this application can integrate NNLF tools on traditional codec platforms, and can be integrated at different filtering positions to achieve collaborative optimization processing of NNLF and ALF, thereby improving video codec performance. The technical solutions of the various embodiments shown above can be used individually or in combination. Furthermore, the technical solutions of the embodiments of this application can be applied to related products such as video codecs, video pre- and post-processing, and video compression.

[0121] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described above.

[0122] Figure 15 A block diagram of a video decoding apparatus according to an embodiment of the present application is shown. The video decoding apparatus can be installed in a device with computing processing capabilities, such as a terminal device or a server.

[0123] Reference Figure 15 As shown, a video decoding apparatus 1500 according to an embodiment of this application includes a decoding unit 1502 and a processing unit 1504.

[0124] The decoding unit 1502 is configured to decode the video bitstream to determine whether the image to be filtered uses a neural network-based loop filter (NNLF) and an adaptive loop filter (ALF) for collaborative optimization. The processing unit 1504 is configured to skip the filtering process related to the filtering result of the NNLF in the ALF if the image to be filtered uses the NNLF and the ALF for collaborative optimization, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

[0125] In some embodiments of this application, based on the foregoing scheme, the decoding unit 1502 is configured to: decode a first flag bit and a second flag bit from the video bitstream, wherein the first flag bit is used to indicate whether the NNLF is enabled and the second flag bit is used to indicate whether the ALF is enabled; if the first flag bit indicates that the NNLF is enabled and the second flag bit indicates that the ALF is enabled, then a third flag bit is decoded from the video bitstream, wherein the third flag bit is used to indicate whether the image to be filtered uses the NNLF; if the third flag bit indicates that the image to be filtered uses the NNLF, then it is determined that the image to be filtered uses the NNLF and the ALF for collaborative optimization processing.

[0126] In some embodiments of this application, based on the foregoing scheme, when the third flag bit is a first value, it indicates that the image to be filtered does not use the NNLF; when the third flag bit is another value, it indicates that the image to be filtered uses the NNLF and indicates the quantization parameter offset corresponding to the image to be filtered.

[0127] In some embodiments of this application, based on the foregoing scheme, when the third flag bit is a second value, it indicates that the image to be filtered uses the NNLF and that the quantization parameter offset corresponding to the image to be filtered is a first offset value; when the third flag bit is a third value, it indicates that the image to be filtered uses the NNLF and that the quantization parameter offset corresponding to the image to be filtered is a second offset value; when the third flag bit is a fourth value, it indicates that some or all blocks of the image to be filtered use the NNLF and that the quantization parameter offset corresponding to the image to be filtered is a third offset value.

[0128] In some embodiments of this application, based on the foregoing scheme, the filtering process related to the filtering result of the NNLF in the ALF is skipped, and the filtering result of the NNLF is used as the processing result of the related filtering process in the ALF, including any of the following processes:

[0129] Skip the coefficient training and filtering process based on the reconstructed image in the ALF, and use the reconstructed image filtered by the NNLF as the processing result of the coefficient training and filtering process based on the reconstructed image in the ALF;

[0130] Skip the coefficient training and filtering process based on the residual image in the ALF, and use the residual image after filtering by the NNLF as the processing result of the coefficient training and filtering process based on the residual image in the ALF;

[0131] Skip the coefficient training and filtering process based on the reconstructed image and residual image in the ALF, and use the reconstructed image and residual image filtered by the NNLF as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in the ALF;

[0132] Skip the coefficient training and filtering process based on the residual image in the ALF, and use the residual image obtained based on the predicted image and the reconstructed image after NNLF filtering as the processing result of the coefficient training and filtering process based on the residual image in the ALF.

[0133] In some embodiments of this application, based on the foregoing scheme, the decoding unit 1502 is further configured to: decode a fourth flag bit from the video bitstream, the fourth flag bit being used to indicate the collaborative optimization method of the NNLF and the ALF, the collaborative optimization method representing the skipped filtering process in the ALF.

[0134] In some embodiments of this application, based on the foregoing scheme, the processing unit 1504 is configured as follows:

[0135] If the fourth flag is the first value, then the coefficient training and filtering process based on the reconstructed image in the ALF is skipped, and the reconstructed image after NNLF filtering is used as the processing result of the coefficient training and filtering process based on the reconstructed image in the ALF.

[0136] If the fourth flag is the second value, then the coefficient training and filtering process based on the residual image in the ALF is skipped, and the residual image after the NNLF filtering is used as the processing result of the coefficient training and filtering process based on the residual image in the ALF.

[0137] If the fourth flag is the third value, then the coefficient training and filtering process based on the reconstructed image and residual image in the ALF is skipped, and the reconstructed image and residual image after NNLF filtering are used as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in the ALF.

[0138] In some embodiments of this application, based on the aforementioned scheme, the input terminal of the NNLF is connected to the input terminal of the deblocking filter DBF to receive the image to be filtered; the output terminal of the DBF is connected to the input terminal of the sample adaptive compensation unit SAO, the output terminal of the SAO is connected to the first input terminal of the ALF, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

[0139] In some embodiments of this application, based on the aforementioned scheme, the input terminal of the NNLF is connected to the input terminal of the DBF to receive the image to be filtered; the output data of the DBF and the output data of the NNLF are fused together as the input data of the SAO, the output terminal of the SAO is connected to the first input terminal of the ALF, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

[0140] In some embodiments of this application, based on the aforementioned scheme, the output data of the DBF and the output data of the NNLF are weighted and fused according to a set weight.

[0141] In some embodiments of this application, based on the foregoing scheme, the input terminal of the DBF is used to receive the image to be filtered, the output terminal of the DBF is connected to the input terminal of the SAO, the output terminal of the SAO is connected to the first input terminal of the ALF and the input terminal of the NNLF respectively, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output data of the ALF and the output data of the NNLF are fused together to form the filtered image.

[0142] In some embodiments of this application, based on the aforementioned scheme, the input terminal of the NNLF is connected to the input terminal of the DBF to receive the image to be filtered; the output terminal of the DBF is connected to the input terminal of the SAO, the output data of the SAO and the output data of the NNLF are fused together and used as the input data of the first input terminal of the ALF, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

[0143] In some embodiments of this application, based on the aforementioned scheme, the input terminal of the DBF is used to receive the image to be filtered, the output terminal of the DBF is connected to the input terminal of the SAO and the input terminal of the NNLF respectively, the output data of the SAO and the output data of the NNLF are fused together and used as the input data of the first input terminal of the ALF, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

[0144] Figure 16 A block diagram of a video encoding apparatus according to an embodiment of the present application is shown. The video encoding apparatus can be installed in a device with computing processing capabilities, such as a terminal device or a server.

[0145] Reference Figure 16 As shown, a video encoding apparatus 1600 according to an embodiment of this application includes: a determining unit 1602, an encoding unit 1604, and a processing unit 1606.

[0146] The determining unit 1602 is configured to determine whether the image to be filtered uses NNLF and ALF for collaborative optimization processing based on the pass-through distortion optimization criterion; the encoding unit 1604 is configured to encode a flag bit to indicate that the image to be filtered uses NNLF if it is determined that the image to be filtered uses NNLF and ALF for collaborative optimization processing; and the processing unit 1606 is configured to skip the filtering process in ALF related to the filtering result of NNLF, and use the filtering result of NNLF as the processing result of the related filtering process in ALF.

[0147] Figure 17 A schematic diagram of a computer system suitable for implementing the computer device of the present application is shown. The computer device may be a video encoding device or a video decoding device as described in the foregoing embodiments.

[0148] It should be noted that, Figure 17 The computer system 1700 of the computer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0149] like Figure 17As shown, the computer system 1700 may include a Central Processing Unit (CPU) 1701, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1702 or programs loaded from storage portion 1708 into Random Access Memory (RAM) 1703, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 1703. The CPU 1701, ROM 1702, and RAM 1703 are interconnected via bus 1704. An Input / Output (I / O) interface 1705 is also connected to bus 1704.

[0150] The following components can be connected to I / O interface 1705: input section 1706 including keyboard, mouse, etc.; output section 1707 including cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; storage section 1708 including hard disk, etc.; and communication section 1709 including network interface card, modem, etc. Communication section 1709 performs communication processing via a network such as the Internet. Drive 1710 is also connected to I / O interface 1705 as needed. Removable media 1711, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1710 as needed so that computer programs read from them can be installed into storage section 1708 as needed.

[0151] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1709, and / or installed from removable medium 1711. When the computer program is executed by central processing unit (CPU) 1701, it performs various functions defined in the system of this application.

[0152] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and a computer program.

[0154] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0155] In another aspect, this application also provides a computer-readable medium, which may be included in the computer device described in the above embodiments; or it may exist independently and not assembled into the computer device. The computer-readable medium carries one or more computer programs, which, when executed by the computer device, cause the computer device to perform the methods described in the above embodiments.

[0156] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0157] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computer device to execute the method according to the embodiments of this application.

[0158] For example, a computer device can be a video decoding device, then the video decoding device can perform... Figure 7 The video decoding method shown; for example, a computer device can be a video encoding device, then the video encoding device can perform... Figure 14 The video encoding method shown.

[0159] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0160] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A video decoding method, characterized in that, include: The video stream is decoded to determine whether the image to be filtered should be co-optimized using a neural network-based loop filter (NNLF) and an adaptive loop filter (ALF). If the image to be filtered is processed using the NNLF and the ALF for collaborative optimization, then the filtering process in the ALF related to the filtering result of the NNLF is skipped, and the filtering result of the NNLF is used as the processing result of the related filtering process in the ALF.

2. The video decoding method according to claim 1, characterized in that, The video stream is decoded to determine whether the image to be filtered should be co-optimized using a neural network-based loop filter (NNLF) and an adaptive loop filter (ALF), including: A first flag bit and a second flag bit are obtained by decoding the video bitstream. The first flag bit is used to indicate whether the NNLF is enabled, and the second flag bit is used to indicate whether the ALF is enabled. If the first flag bit indicates that the NNLF is enabled and the second flag bit indicates that the ALF is enabled, then a third flag bit is decoded from the video bitstream. The third flag bit is used to indicate whether the image to be filtered uses the NNLF. If the third flag indicates that the image to be filtered uses the NNLF, then it is determined that the image to be filtered uses the NNLF and the ALF for collaborative optimization processing.

3. The video decoding method according to claim 2, characterized in that, When the third flag is the first value, it indicates that the image to be filtered does not use the NNLF; When the third flag bit is any other value, it indicates that the image to be filtered uses the NNLF and indicates the quantization parameter offset corresponding to the image to be filtered.

4. The video decoding method according to claim 3, characterized in that, When the third flag bit is any other value, it indicates that the image to be filtered uses the NNLF and indicates the quantization parameter offset corresponding to the image to be filtered, including: When the third flag bit is the second value, it indicates that the image to be filtered uses the NNLF and that the quantization parameter offset corresponding to the image to be filtered is the first offset value; When the third flag bit is a third value, it indicates that the image to be filtered uses the NNLF, and the quantization parameter offset corresponding to the image to be filtered is a second offset value; When the third flag is the fourth value, it indicates that the NNLF is used for some or all blocks of the image to be filtered, and the quantization parameter offset corresponding to the image to be filtered is the third offset value.

5. The video decoding method according to claim 1, characterized in that, Skip the filtering process in the ALF that is related to the filtering result of the NNLF, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF, including any of the following processes: Skip the coefficient training and filtering process based on the reconstructed image in the ALF, and use the reconstructed image filtered by the NNLF as the processing result of the coefficient training and filtering process based on the reconstructed image in the ALF; Skip the coefficient training and filtering process based on the residual image in the ALF, and use the residual image after filtering by the NNLF as the processing result of the coefficient training and filtering process based on the residual image in the ALF; Skip the coefficient training and filtering process based on the reconstructed image and residual image in the ALF, and use the reconstructed image and residual image filtered by the NNLF as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in the ALF; Skip the coefficient training and filtering process based on the residual image in the ALF, and use the residual image obtained based on the predicted image and the reconstructed image after NNLF filtering as the processing result of the coefficient training and filtering process based on the residual image in the ALF.

6. The video decoding method according to claim 1, characterized in that, The video decoding method further includes: The fourth flag bit is obtained by decoding from the video bitstream. The fourth flag bit is used to indicate the cooperative optimization method of the NNLF and the ALF. The cooperative optimization method represents the skipped filtering process in the ALF.

7. The video decoding method according to claim 6, characterized in that, If the fourth flag is the first value, then the coefficient training and filtering process based on the reconstructed image in the ALF is skipped, and the reconstructed image after NNLF filtering is used as the processing result of the coefficient training and filtering process based on the reconstructed image in the ALF. If the fourth flag is the second value, then the coefficient training and filtering process based on the residual image in the ALF is skipped, and the residual image after the NNLF filtering is used as the processing result of the coefficient training and filtering process based on the residual image in the ALF. If the fourth flag is the third value, then the coefficient training and filtering process based on the reconstructed image and residual image in the ALF is skipped, and the reconstructed image and residual image after NNLF filtering are used as the processing result of the coefficient training and filtering process based on the reconstructed image and residual image in the ALF.

8. The video decoding method according to any one of claims 1 to 7, characterized in that, The input of the NNLF is connected to the input of the deblocking filter DBF to receive the image to be filtered; the output of the DBF is connected to the input of the sample adaptive compensation unit SAO, the output of the SAO is connected to the first input of the ALF, the output of the NNLF is connected to the second input of the ALF, and the output of the ALF is used to output the filtered image.

9. The video decoding method according to any one of claims 1 to 7, characterized in that, The input terminal of the NNLF is connected to the input terminal of the DBF to receive the image to be filtered; the output data of the DBF and the output data of the NNLF are fused together as the input data of the SAO, the output terminal of the SAO is connected to the first input terminal of the ALF, the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

10. The video decoding method according to claim 9, characterized in that, The output data of the DBF and the output data of the NNLF are weighted and fused according to the set weights.

11. The video decoding method according to any one of claims 1 to 7, characterized in that, The input terminal of the DBF is used to receive the image to be filtered. The output terminal of the DBF is connected to the input terminal of the SAO. The output terminal of the SAO is connected to the first input terminal of the ALF and the input terminal of the NNLF. The output terminal of the NNLF is connected to the second input terminal of the ALF. The output data of the ALF and the output data of the NNLF are fused together to form the filtered image.

12. The video decoding method according to any one of claims 1 to 7, characterized in that, The input terminal of the NNLF is connected to the input terminal of the DBF to receive the image to be filtered; the output terminal of the DBF is connected to the input terminal of the SAO, and the output data of the SAO and the output data of the NNLF are fused together as the input data of the first input terminal of the ALF; the output terminal of the NNLF is connected to the second input terminal of the ALF, and the output terminal of the ALF is used to output the filtered image.

13. The video decoding method according to any one of claims 1 to 7, characterized in that, The input terminal of the DBF is used to receive the image to be filtered. The output terminal of the DBF is connected to the input terminal of the SAO and the input terminal of the NNLF respectively. The output data of the SAO and the output data of the NNLF are fused together and used as the input data of the first input terminal of the ALF. The output terminal of the NNLF is connected to the second input terminal of the ALF. The output terminal of the ALF is used to output the filtered image.

14. A video encoding method, characterized in that, include: The rate-distortion optimization criterion is used to determine whether the image to be filtered should be processed using NNLF and ALF for collaborative optimization. If it is determined that the image to be filtered uses the NNLF and the ALF for collaborative optimization processing, then a flag bit is encoded to indicate that the image to be filtered uses the NNLF; Skip the filtering process in the ALF that is related to the filtering result of the NNLF, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

15. A video decoding device, characterized in that, include: The decoding unit is configured to decode the video bitstream to determine whether the image to be filtered is to be co-optimized using a neural network-based loop filter NNLF and an adaptive loop filter ALF. The processing unit is configured to skip the filtering process related to the filtering result of the NNLF in the ALF if the image to be filtered is subjected to collaborative optimization processing using the NNLF and the ALF, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

16. A video encoding apparatus, characterized in that, include: The determination unit is configured to use the pass-rate distortion optimization criterion to determine whether the image to be filtered should be processed using NNLF and ALF for collaborative optimization. The encoding unit is configured to encode a flag bit to indicate that the image to be filtered uses the NNLF if it is determined that the image to be filtered uses the NNLF for collaborative optimization processing with the NNLF and the ALF. The processing unit is configured to skip the filtering process in the ALF that is related to the filtering result of the NNLF, and use the filtering result of the NNLF as the processing result of the related filtering process in the ALF.

17. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 14.

18. A computer device, characterized in that, include: One or more processors; A memory for storing one or more computer programs that, when executed by one or more processors, cause the computer device to perform the method of any one of claims 1 to 14.

19. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, wherein a processor of a computer device reads from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method of any one of claims 1 to 14.

20. A method for storing video streams, characterized in that, The video stream is decoded by the video decoding method according to any one of claims 1 to 13, or the video stream is generated by the video encoding method according to claim 14.

Images