Nn-based in loop filter architectures with separable convolution and switching order of decomposition

The neural network-based filtering process with optimized separable convolutions and decomposition orders addresses inefficiencies in video decoding, improving performance and resource utilization.

HK40135692APending Publication Date: 2026-07-31QUALCOMM INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
HK62026123532
Authority / Receiving Office
HK · HK
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-06-10
Filing Date
2026-05-18
Publication Date
2026-07-31

AI Technical Summary

Technical Problem

Existing video decoding technologies face inefficiencies in processing video data due to the complexity of neural network-based filtering processes, particularly in handling separable convolutions and decomposition orders, which impact decoding performance and resource utilization.

Method used

A neural network-based filtering process is implemented with a specific architecture that includes a first and second backbone block process, each comprising a convolution operation with different dimensions (MxN and NxM), enhancing the decoding efficiency by optimizing the separable convolution and decomposition order.

Benefits of technology

The proposed architecture improves decoding efficiency and resource utilization by optimizing separable convolutions and decomposition orders, leading to enhanced video decoding performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

An apparatus for decoding video data receives a picture of the video data; reconstructing a block of a picture of the video data to generate a reconstructed block; applying a neural network (NN)-based filtering process to the reconstructed block to generate a filtered block, where the NN-based filtering process comprises a first backbone block process followed by a second backbone block process, where the first backbone block process comprises a first MxN convolution followed by a first NxM convolution, and the second backbone block process comprises a second NxM convolution followed by a second MxN convolution, wherein N and M are different integer values; determining a decoded block of the video data based on the filtered block; and outputting a decoded version of the picture, wherein the decoded version of the picture comprises a decoded block of the video data.
Need to check novelty before this filing date? Find Prior Art

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202480036577.X (22) Application Date 2024.06.11 (30) Priority Data 63 / 507,706 2023.06.12 US 18 / 738,612 2024.06.10 US (85) PCT International Application Entering National Phase Date 2025.12.01 (86) PCT International Application Application Data PCT / US2024 / 033409 2024.06.11 (87) PCT International Application Publication Data WO2024 / 258851 EN 2024.12.19 (71) Applicant: Qualcomm Incorporated, California, USA (72) Inventors: Y. Lee, D. Rusanovsky, M. Karl Czerwitz (74) Patent Agency: Beijing Liu Shen Law Firm, 11105 Patent Attorney: An Zhifei (51) Int.Cl. H04N 19 / 42 (2006.01) G06N 3 / 0464 (2006.01) H04N 19 / 82 (2006.01) (54) Invention Title: NN-based In-Loop Filter Architecture with Separable Convolution and Exchanged Decomposition Order (57) Abstract: An apparatus for decoding video data receives a picture of video data; reconstructs blocks of the picture of the video data to generate a reconstructed block; applies a neural network (NN)-based filtering process to the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes a first backbone block process followed by a second backbone block process, wherein the first backbone block process includes a first MxN convolution followed by a first NxM convolution, and the second backbone block process includes a second NxM convolution followed by a second MxN convolution, wherein N and M are different integer values; determines a decoded block of video data based on the filtered block; and outputs a decoded version of the picture, wherein the decoded version of the picture includes the decoded block of the video data. Claims: 3 pages Description: 33 pages Drawings: 26 pages CN 121220037 A 2025.12.26 CN 1 21 22 00 37 A

Claims

1. A method for decoding video data, the method comprising: Images that receive video data; Reconstruct the blocks of the images in the video data to generate reconstructed blocks; A neural network (NN) based filtering process is applied to the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes a first backbone block process followed by a second backbone block process, wherein the first backbone block process includes a first MxN convolution followed by a first NxM convolution, and the second backbone block process includes a second NxM convolution followed by a second MxN convolution, where N and M are different integer values; Determine the decoding block of the video data based on the filtered block; as well as Output a decoded version of the image, wherein the decoded version of the image includes the decoded blocks of the video data.

2. The method according to claim 1, wherein M equals 3 and N equals 1.

3. The method of claim 1, wherein the first backbone block process includes a first 1x1 convolution parallel to the first MxN convolution and the first NxM convolution, and the second backbone block process includes a second 1x1 convolution parallel to the second NxM convolution and the second MxN convolution.

4. The method of claim 3, wherein the first backbone process includes a first parametric rectified linear unit (PReLU) layer to receive a first set of outputs from the first 1x1 convolution and the first NxM convolution, and the second backbone process includes a second PReLU layer to receive a second set of outputs from the second 1x1 convolution and the second MxN convolution.

5. The method of claim 4, wherein the first backbone block process further includes a third 1x1 convolution followed by a third MxN convolution followed by a third NxM convolution, and the second backbone block process further includes a fourth 1x1 convolution followed by a fourth NxM convolution followed by a fourth MxN convolution.

6. The method of claim 1, wherein applying the NN-based filtering process to the reconstructed block to generate the filtered block comprises applying a first instance of the first backbone block process followed by a first instance of the second backbone block process, followed by a second instance of the first backbone block process, followed by a second instance of the second backbone block process.

7. The method of claim 1, wherein the decoding method is performed as part of a video encoding process.

8. An apparatus for decoding video data, the apparatus comprising: A memory configured to store video data; One or more processors, said one or more processors being implemented in a circuit and configured to: Images that receive video data; Reconstruct the blocks of the images in the video data to generate reconstructed blocks; A neural network (NN) based filtering process is applied to the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes a first backbone block process followed by a second backbone block process, wherein the first backbone block process includes a first MxN convolution followed by a first NxM convolution, and the second backbone block process includes a second NxM convolution followed by a second MxN convolution, where N and M are different integer values; Determine the decoding block of the video data based on the filtered block; as well as Output a decoded version of the image, wherein the decoded version of the image includes the decoded blocks of the video data.

9. The device according to claim 8, wherein M equals 3 and N equals 1.

10. The apparatus of claim 8, wherein the first backbone process includes a first 1x1 convolution parallel to the first MxN convolution and the first NxM convolution, and the second backbone process includes a second 1x1 convolution parallel to the second NxM convolution and the second MxN convolution.

11. The apparatus of claim 10, wherein the first backbone process includes a first parametric rectified linear unit (PReLU) layer to receive a first set of outputs from the first 1x1 convolution and the first NxM convolution, and the second backbone process includes a second PReLU layer to receive a second set of outputs from the second 1x1 convolution and the second MxN convolution.

12. The apparatus of claim 11, wherein the first backbone process further comprises a third 1x1 convolution followed by a third MxN convolution followed by a third NxM convolution, and the second backbone process further comprises a fourth 1x1 convolution followed by a fourth NxM convolution followed by a fourth MxN convolution.

13. The apparatus of claim 8, wherein applying the NN-based filtering process to the reconstructed block to generate the filtered block comprises applying a first instance of the first backbone block process followed by a first instance of the second backbone block process, followed by a second instance of the first backbone block process followed by a second instance of the second backbone block process.

14. The device of claim 8, further comprising a display configured to display decoded video data.

15. The device of claim 8, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

16. An apparatus for encoding video data, the apparatus comprising: A memory configured to store video data; One or more processors, said one or more processors being implemented in a circuit and configured to: Determine the reconstructed blocks of images in the video data; A neural network (NN) based filtering process is applied to the reconstructed block to generate a filtered block, wherein the NN-based filtering process includes a first backbone block process followed by a second backbone block process, wherein the first backbone block process includes a first MxN convolution followed by a first NxM convolution, and the second backbone block process includes a second NxM convolution followed by a second MxN convolution, where N and M are different integer values; Determine the decoding block of the video data based on the filtered block; as well as The decoded version of the image is stored, wherein the decoded version of the image includes the decoded block of the video data.

17. The device of claim 16, wherein M equals 3 and N equals 1.

18. The apparatus of claim 16, wherein the first backbone process includes a first 1x1 convolution parallel to the first MxN convolution and the first NxM convolution, and the second backbone process includes a second 1x1 convolution parallel to the second NxM convolution and the second MxN convolution.

19. The apparatus according to claim 18, wherein: The first backbone block process includes a first parametric rectified linear unit (PReLU) layer to receive a first set of outputs from the first 1x1 convolution and the first NxM convolution, and includes a third 1x1 convolution followed by a third MxN convolution followed by a third NxM convolution. The second backbone block process includes a second PReLU layer to receive a second set of outputs from the second 1x1 convolution and the second MxN convolution, and includes a fourth 1x1 convolution followed by a fourth NxM convolution followed by a fourth MxN convolution.

20. The apparatus of claim 16, wherein applying the NN-based filtering process to the reconstructed block to generate the filtered block comprises applying a first instance of the first backbone block process followed by a first instance of the second backbone block process, followed by a second instance of the first backbone block process followed by a second instance of the second backbone block process.