Post-processing filter for video compression of multi-scale feature representation based on convolutional neural network
Through the CNN framework based on wavelet decomposition, the step-by-step subband network and hybrid enhancement network are used to solve the problems of information loss and compression distortion in video compression, and the reconstruction performance and compression effect of video images are significantly improved.
Patent Information
- Application Number
- CN202280094621.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-13
- Publication Date
- 2024-11-19
AI Technical Summary
Existing video compression methods lead to information loss and compression distortion during encoding, such as color blocks, softening and ribbons, especially at high compression ratios, and deep learning-based methods fail to make full use of residual density. The information correlation between blocks and different frequencies limits the learning and feature selection capabilities.
The convolutional neural network (CNN) framework based on wavelet decomposition, including step-by-step subband network and hybrid enhancement network, decompose input images through discrete wavelet transformation, and use technologies such as Res2Net module and hollow convolution to recover and expand the wavelet subband. Reduce information loss and improve image quality.
It significantly improves the reconstruction performance of video images, reduces the occurrence of compression distortion, and retains more process texture details and visible details. Compared with traditional methods, it has significantly improved compression performance.
Smart Images

Figure CN118985002A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to video compression schemes that can improve reconstruction performance. More specifically, the present disclosure is directed to systems and methods for providing a convolutional neural network filter that post-processes video compression based on wavelet decomposition. Background Art
[0002] Common image and video compression methods include those using the Joint Photographic Experts Group (JPEG) standard (e.g., for still images) and the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) standards (e.g., for video). In these methods, quantization and prediction processes are performed during the encoding process, resulting in irreversible information loss and various compression artifacts in the compressed image / video, such as blocking, blurring, and banding. This disadvantage is particularly evident when a high compression ratio is used.
[0003] To address the above shortcomings, a variety of deep learning based methods have been used. These methods include frameworks / networks based on simple concatenated layers, deep residual blocks, dense connections, cascade connections, and feature reuse. However, most of these methods do not adopt advanced features such as residual dense blocks and information correlation between different frequencies. Therefore, these methods cannot further improve their learning and feature selection capabilities, resulting in very limited effects in removing compression artifacts. Summary of the invention
[0004] The present disclosure relates to systems and methods for improving video image quality using a neural network based on wavelet decomposition for video compression (e.g., video processing). Although the following systems and methods are described as being related to video processing, in some embodiments, these systems and methods may also be used in other image processing systems and methods. A convolutional neural network (CNN) framework can be trained by deep learning and / or artificial intelligence schemes. The present disclosure provides a CNN framework for improving the image quality of an input image based on two subnetworks: a stepped subband network and a hybrid enhancement network. The stepped subband network includes a Res2Net group (R2NG) composed of Res2Net modules for representing multi-scale features. The hybrid enhancement network uses dilated convolutions and standard convolutions for an extended receptive field while avoiding blind spots, which is different from using only dilated convolutions. By using these two subnetworks, the CNN framework has improved image reconstruction performance compared to common reconstruction systems and methods.
[0005] In some embodiments, the method uses a CNN framework for image compression. The method receives an input image, which in some embodiments may be a video frame. The method decomposes the input image into a set of wavelet subbands using discrete wavelet transform. The method inputs the set of wavelet subbands into the CNN framework. The CNN framework includes two subnetworks. The first subnetwork (or stepped subband network) is configured to restore the set of wavelet subbands from high frequency to low frequency, and use the restored high-frequency wavelet subbands to restore the low-frequency wavelet subbands. The second subnetwork (or hybrid enhancement network) is configured to use hybrid convolution to expand the size of the receptive field of the signal of the restored set of wavelet subbands. The method receives an enhanced version of the input image from the CNN framework.
[0006] In some embodiments, the method may be implemented by a tangible medium, a non-transitory mechanism, or a computer-readable medium having processor instructions stored thereon. When the processor instructions are executed by one or more processors, the one or more processors are caused to perform one or more aspects / features of the method described herein. In other embodiments, the method may be implemented by a system including a computer processor and a non-transitory computer-readable storage medium storing instructions. When the instructions are executed by a computer processor, the computer processor is caused to perform one or more operations of the method described herein.
[0007] In some embodiments, the method can be implemented in various data processing flows such as loop filtering modules of video or image codecs. The method can also be implemented in data processing flows such as post filtering modules of video or image codecs.
[0008] In some embodiments, the method can be implemented by a system having an encoder and a decoder. In such embodiments, the encoder can encode the input video / picture based on a set of rules (e.g., a video codec) and then transmit the encoded video / picture. After receiving the encoded video, the decoder can decode the encoded video based on the set of rules and then generate the decoded video as the output video. In some embodiments, there may be some "post filters" or "post filtering" processes to further process the output video.
[0009] Examples of post filters include the CNN-based post-processing filters discussed in this article. For example, the post filter can be connected to the output of the decoder and use the output video as the input of the post filter to further process / filter the output video.
[0010] In some embodiments, the CNN-based filters discussed herein can be "loop filters" in encoders and decoders. For example, in such an embodiment, the filter can be in a loop filtering module that is configured to enhance the quality of a reconstructed picture or a region of a picture (e.g., a coding unit, a coding tree unit, a sub-picture, etc.). In such an embodiment, the reconstructed picture at the encoder or decoder can be used as an input to the loop filtering module. In some embodiments, the output of the loop filtering module can be connected to a buffer (e.g., a decoded picture buffer). BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following briefly describes the drawings. The drawings only show certain aspects or embodiments of the present disclosure, and those skilled in the art can deduce other drawings from these drawings without creative work.
[0012] Figure 1 is a schematic diagram illustrating a convolutional neural network (CNN) framework according to one or more embodiments of the present disclosure.
[0013] Figure 2A is a schematic diagram showing a Res2Net group in a CNN framework according to one or more embodiments of the present disclosure.
[0014] Figure 2B is a schematic diagram showing a Res2Net module of a CNN framework according to one or more embodiments of the present disclosure.
[0015] Figure 3 is a schematic diagram of a mixed convolution group of a CNN framework according to one or more embodiments of the present disclosure.
[0016] Figure 4 is a schematic diagram of a wireless communication system according to one or more embodiments of the present disclosure.
[0017] Figure 5 is an exemplary block diagram of a terminal device according to one or more embodiments of the present disclosure.
[0018] Figure 6 is a flow chart of a method according to one or more embodiments of the present disclosure.
[0019] Figure 7 is a flow chart of a video encoder according to one or more embodiments of the present disclosure.
[0020] Figure 8 is a flow chart of a video decoder according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION
[0021] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following briefly describes the drawings. The drawings only show some aspects or embodiments of the present disclosure, and ordinary technicians in this field can still deduce other drawings from these drawings without creative work.
[0022] Figure 1 1 is a schematic diagram illustrating a convolutional neural network (CNN) framework 100 according to one or more embodiments of the present disclosure. The CNN framework 100 is configured to learn, train, and / or use residual information to improve the image quality of an input image 101. As discussed in detail below, the CNN framework 100 uses two types of sub-networks including a stepped sub-band network 102 and a hybrid enhancement network 103 to reconstruct the input image. In some embodiments, the CNN framework 100 may include Figure 1 One or more additional filters or convolution filters not shown.
[0023] The CNN framework 100 receives an input image 101. The input image 101 may be a single image that has been compressed before transmission, or one of a set of video frames from a video. The CNN framework may receive the input image 101 from a computing device connected to the CNN framework or a system employing the CNN framework through a network or a direct connection. The signal of the input image 101 may be represented by wavelet coefficients after applying a discrete wavelet transform (DWT) 106. The wavelet coefficients reflect the similarity between the wavelet basis and the image signal and form wavelet subbands. The CNN framework 100 decomposes the input image 100 into four wavelet subbands using a discrete wavelet transform (DWT) 106: a low-frequency feature (LL) subband 107A, a vertical feature (HL) subband 107B, a horizontal feature (LH) subband 107C, and a diagonal feature (HH) subband 107D. Most of the content from the input image 101 is concentrated in the LL subband 107A. The HL subband 107B, the LH subband 107C, and the HH subband 107D are high frequency components of the input image 101 and contain information of regions in the input image 101 where grayscale values change dramatically (e.g., edges and textures in the input image 101). The HH subband 107D includes the least amount of information from the input image 107D. Each wavelet subband 107 represents a different direction, so that the relationship between the wavelet subbands 107 includes feature locations from the input image 101. The CNN framework 100 processes the wavelet subbands 107 in the following order: HH, HL, LH, and LL or HH, LH, HL, and LL. HH is processed first because HH contains the least information of the wavelet subband 107 and is easily lost during the compression process. LL is processed last because LL contains the most information and less information is lost during the compression process.
[0024] The CNN framework 100 inputs each wavelet subband 107 through 1×1 convolutional layers 113 and is gradually input to the stepped subband network 102. The stepped subband network 102 processes the high-frequency subbands 107B-D before the low-frequency subband 107A, and uses the restored high-frequency subbands 107B-D to help restore the low-frequency subband 107A. In some embodiments, the stepped subband network 102 processes the wavelet subbands 107 one by one from the highest frequency to the lowest frequency, and uses the processed wavelet subbands 107 to help restore the subsequently processed wavelet subbands. Each wavelet subband 107 corresponds to a Res2Net group 105 in the stepped subband network 102. Figure 2A-2B The Res2Net group 105 is further described.
[0025] The ladder subband network can be represented by equations 1 to 4, where LL, HL, LH, HH represent input wavelet subbands, LL′, HL′, LH′, HH′ represent outputs from the ladder subband network for each wavelet subband 107, and R LL , R HL , R LH and R HH Res2Net groups 105 corresponding to each wavelet subband 107 are represented respectively.
[0026] HH′=R HH (HH) Formula 1
[0027] LH′=R LH (LH+HH′) Formula 2
[0028] HL′=R HL (HL+LH′) Formula 3
[0029] LL′=R LL (LL+HL′) Formula 4
[0030] The CNN framework outputs each wavelet subband from the Res2Net group 105 through a 1×1 convolution layer 113, and performs an inverse discrete wavelet transform (IDWT) 108 on the wavelet subband 107 to reconstruct the signal from the input image 101. The CNN framework 100 inputs the reconstructed signal to the hybrid enhancement network 103 to expand the size of the receptive field of the reconstructed signal. The hybrid enhancement network 103 uses a combination of dilated convolution and standard convolution in each hybrid convolution group 109, and also includes two 3×3 convolution layers 111, one of which is at the beginning of the hybrid enhancement network 103 and the other is at the end of the hybrid enhancement network 103 (e.g., after the hybrid convolution group 109). Figure 3Also depicted is the structure of the hybrid convolutional group 109. The hybrid enhancement network 103 outputs a signal representing an enhanced version of the input image 101 (eg, enhanced image 104).
[0031] Figure 2A is a schematic diagram showing a Res2Net group in a CNN framework according to one or more embodiments of the present disclosure. Figure 2A As shown, the Res2Net group 105 includes a 3×3 convolutional layer 203A at the beginning and a 3×3 convolutional layer 203B at the end. In some embodiments, the Res2Net group may include an additional 3×3 convolutional layer 203 at its residual connection. The Res2Net group 105 also includes a set of Res2Net modules 201A-B. In some embodiments, the Res2Net group 105 may include three Res2Net modules 201. In other embodiments, any number of Res2Net modules 201 may be included in each Res2Net group 105. With similar computational load, each Res2Net module has stronger multi-feature extraction capabilities than traditional bottleneck blocks. The use of Res2Net modules 201 in the Res2Net group improves the multi-scale representation capability of CNN.
[0032] Figure 2B is a schematic diagram showing a Res2Net module 201. The Res2Net module 201 processes segmentation in a multi-scale manner, extracting global and local information from the wavelet subbands 107. Each segment 209 in the Res2Net module 201 is connected to other segments and passed through a 3×3 convolution layer 203 to fuse information at different scales from the input image 101. The Res2Net module 201 includes a 1×1 convolution layer 205A at its beginning and a 1×1 convolution layer 205B after the 3×3 convolution layer 203 and before the spatial attention and channel attention module (SA&CA) 207, which is embedded to adaptively enhance the channel and spatial feature responses of the Res2Net module 201.
[0033] Figure 3is a schematic diagram of a hybrid convolution group 103 of a hybrid enhancement network 103 according to one or more embodiments of the present disclosure. The hybrid convolution network 103 uses a set of hybrid convolution groups 109 to expand the size of the receptive field of the reconstructed signal 301 from the input image 101. In some embodiments, the hybrid enhancement network 103 includes three hybrid convolution groups 109 and two 3×3 convolution layers 111. In other embodiments, the hybrid enhancement network 1034 includes any number of hybrid convolution groups 109. Each hybrid convolution group 109 uses a combination of dilated convolution and standard convolution. The dilated convolution expands the receptive field without increasing the parameters of the reconstructed signal or reducing the resolution of the reconstructed signal. Due to the predefined gap of the dilated convolution, using only the dilated convolution will result in blind spots in the receptive field because there is a lack of available contextual information between the pixels of the input image 101, so combining the dilated convolution with the standard convolution can produce an improved output 303.
[0034] like Figure 3 As shown, the mixed convolution group 109 receives the input 301. The mixed convolution layer of the mixed enhancement network 103 includes N channels (e.g., 64 channels), where “ PX N" channels are generated by dilated convolutions, while the remaining channels are generated by standard convolutions (e.g., p is the convolution coefficient). For the first hybrid convolution group 109A in the hybrid enhancement network 103, the input 301 is the reconstructed signal after processing by the IDWT 108. The next hybrid convolution group 109B receives the output 303 of the first hybrid convolution group 109A as input 301. The reconstructed signal is input and output to the hybrid enhancement network 103 from the sequence of hybrid convolution groups 109 in a similar manner. The output of the last hybrid convolution group 109C is a signal representing the enhanced image 104.
[0035] Figure 3 The hybrid convolution group 109 includes a set of densely connected hybrid convolution blocks 305 between two 3×3 convolution layers 111. The dilation coefficients of the set of hybrid convolution blocks 305 are 1, 2, 4, and 8, respectively. The hybrid convolution group 109 can be represented by equations 5-9, where M d=n represents a hybrid convolution block 305 with a hole coefficient of n, X n represents the nth output of the mixed convolution group 109, and X n Represents a combination of n mixed convolution groups 109. The "fusion" function is used to combine different outputs of the mixed convolution group 109.
[0036] X 1 = M d=1 (X n-1 ) Formula 5
[0037] X 2 = M d=2 (Md=1 (X {n-1} )) Formula 6
[0038] X 3 = M d=4 (M d=2 (M d=1 (X n-1 ))) Formula 7
[0039] X 4 = M d=8 (M d=4 (M d=2 (M d=1 (X n-1 )))) Formula 8
[0040] X n = f 融合 ([X 1 ,X 2 ,X 3 ,X 4 ]) Formula 9
[0041] A set of comparative tests showed that the CNN framework restored objects more effectively than VTM 11.0-NNVC and preserved more process texture details. Many process improvements were found in areas of the test video, including the wall in the background, the man's clothes, and the man's silhouette. In another set of comparative tests, it was demonstrated that the CNN framework eliminated block effects. For example, it was observed that the CNN framework preserved the visible details of the necklace in the test video more effectively than VTM 11.0-NNVC.
[0042] Tables 1-3 below show quantitative measurements of using the CNN framework 100 as a post-processing filter on the input image 101 compared to VTM 11.0-NNVC. In Tables 1-3, Y-PNSR represents the peak signal-to-noise ratio of the Y channel, and Y-MSIM represents the multi-scale structural similarity of the Y channel of the processed image. In addition, the negative BD rate value represents the coding gain of using the CNN framework 100.
[0043] Table 1 shows quantitative measurements of the BD rate using the CNN framework in a random access (RA) configuration compared to VTM 11.0-NNVC.
[0044] Table 1
[0045]
[0046]
[0047] Table 2 shows the quantitative measurements of using the CNN framework in all intra-frame (AI) configurations compared to VTM 11.0-NNVC.
[0048] Table 2
[0049] Metrics Y-PSNR Y-MSIM Category B -3.17% -2.93% Category C -4.31% -3.39% Category D -4.58% -3.18% Category E -5.51% -6.29%
[0050] Table 3 shows the quantitative measurements of using the CNN framework in low-latency prediction frames (LDP) compared to VTM 11.0-NNVC.
[0051] Table 3
[0052] Metrics Y-PSNR Y-MSIM Category B -3.19% -2.63% Category C -3.62% -2.65% Category D -4.15% -2.73% Category E -5.74% -6.11%
[0053] These results show that the CNN framework 100 reduces the average BD rate by 2.99%, 4.8%, 3.72%, and 4.5% relative to VTM 11.0-NNVC for the Y channel of Class B, Class C, Class D, and Class E in AI, RA, and LDP configurations, respectively. Therefore, the CNN framework 100 produces improved performance in terms of compression relative to VTM11.0-NNVC.
[0054] Figure 4 4 is a schematic diagram of a wireless communication system 400 according to one or more embodiments of the present disclosure. The wireless communication system 400 may implement the CNN framework 100 discussed herein. Figure 4 As shown, the wireless communication system 400 may include a network device (or base station) 401. Examples of the network device 401 include a base transceiver station (BTS), a node B (NB), an evolved node B (eNB or eNodeB), a next generation node B (gNB or gNodeB), a wireless fidelity (Wi-Fi) access point (AP), etc. In some embodiments, the network device 401 may include a relay station, an access point, a vehicle-mounted device, a wearable device, etc. The network device 401 may include a wireless connection device for a communication network, such as: a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a wideband CDMA (WCDMA) network, a long-term evolution (LTE) network, a cloud radio access network (CRAN), a network based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 (e.g., a Wi-Fi network), an Internet of Things (IoT) network, a device-to-device (D2D) network, a next generation network (e.g., a 5G network), a future evolved public land mobile network (PLMN), etc. A 5G system or network may be referred to as a New Radio (NR) system or network.
[0055] exist Figure 4, the wireless communication system 400 also includes a terminal device 403. The terminal device 403 may be an end-user device configured to facilitate wireless communication. The terminal device 403 may be configured to wirelessly connect to the network device 401 (e.g., via a wireless channel 405) according to one or more corresponding communication protocols / standards. The terminal device 403 may be mobile or fixed. The terminal device 403 may be a user equipment (UE), an access terminal, a user unit, a user station, a mobile site, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent, or a user device. Examples of the terminal device 403 include a modem, a cellular phone, a smart phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, an Internet of Things (IoT) device, a device used in a 5G network, a device used in a public land mobile network, and the like. For ease of illustration, Figure 4 Only one network device 401 and one terminal device 403 are shown in the wireless communication system 400. However, in some cases, the wireless communication system 400 may include additional network devices 401 and / or terminal devices 403.
[0056] Figure 5 4 is an exemplary block diagram of a terminal device 403 (e.g., which can implement the methods discussed herein) according to one or more embodiments of the present disclosure. As shown in the figure, the terminal device 403 includes a processing unit 510 (e.g., DSP, CPU, GPU, etc.) and a memory 520. The processing unit 510 can be configured to implement Figure 6Instructions corresponding to the method 800 and / or other aspects of the above-mentioned embodiments. It should be understood that the processor 510 in the implementation of the present technology can be an integrated circuit chip and has signal processing capabilities. In the implementation process, the steps in the aforementioned method can be implemented by using hardware integrated logic circuits or software instructions in the processor 510. The processor 510 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present technology can be implemented or executed. The general-purpose processor 510 can be a microprocessor, or the processor 510 can be any conventional processor, etc. instead. The steps in the method disclosed with reference to the implementation of the present technology can be directly executed or completed by a decoding processor implemented as hardware, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory 520, and the processor 510 reads the information in the memory 520 and completes the steps in the aforementioned method in combination with its hardware.
[0057] It is understood that the memory 520 in the implementation of the present technology can be a volatile memory or a non-volatile memory, or can include both volatile memory and non-volatile memory. Non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. Volatile memory can be a random access memory (RAM) and used as an external cache. As an exemplary and non-limiting description, many forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct Rambus random access memory (DRRAM). It should be noted that the memory in the system and method described herein is intended to include but is not limited to these memories and any other suitable type of memory. In some embodiments, the memory can be a non-temporary computer-readable storage medium that stores instructions that can be executed by a processor.
[0058] Figure 6is a flow chart of a method according to one or more embodiments of the present disclosure. Method 800 can be implemented by a system (e.g., a system including CNN framework 100). Method 800 is used to enhance image quality (particularly for compressed images). Method 800 includes receiving an input image 101 at box 601. At box 603, method 800 continues by decomposing the input image 101 into a set of wavelet subbands 107. In some embodiments, the decomposition of the input image is completed using DWT 106. At box 605, method 800 continues by inputting the wavelet subbands 107 into the CNN framework 100. CNN framework 100 includes two subnetworks. The first subnetwork (e.g., Figure 2A and 2B The stepped subband network 102 in FIG. 1 is configured to restore the set of wavelet subbands 107 from high frequency to low frequency, and restore the low frequency subband 107A using the restored high frequency wavelet subbands 107B-D. The second subnetwork (e.g., Figure 3 The hybrid enhancement network 103 is configured to use hybrid convolution to expand the size of the receptive field of the recovered set of wavelet subbands. In some embodiments, the CNN framework 100 can apply the IDWT 108 to the recovered set of wavelet subbands between the first subnetwork and the second subnetwork to create a reconstructed signal. At block 607, the method 800 receives an enhanced version of the input image 101 from the CNN framework 100.
[0059] In some embodiments, the first sub-network may include one or more Res2Net groups 105, wherein each Res2Net group 105 includes one or more Res2Net modules 201. In further embodiments, each Res2Net group 105 may include a first 3×3 convolutional layer 203A at the beginning of the Res2Net group 105 and a second 3×3 convolutional layer 203B at the end of the Res2Net group 105. In some embodiments, each Res2Net module 201 may include one or more 3×3 convolutional layers 203 between the first 1×1 convolutional layer 205A and the second 1×1 convolutional layer 205B. In further embodiments, each Res2Net module 210 may include a spatial attention and channel attention module 207 implemented after the second 1×1 convolutional layer 205B.
[0060] In some embodiments, the hybrid convolution may include a combination of a dilated convolution and a standard convolution. In further embodiments, the second subnetwork may include one or more hybrid convolution groups between the first 3×3 convolution layer and the second 3×3 convolution layer, and each hybrid convolution group may include two or more densely connected hybrid convolution blocks. In an example embodiment, each hybrid convolution group includes four hybrid convolution blocks, whose dilated coefficients are 1, 2, 4, and 8, respectively.
[0061] Figure 7 is an exemplary block diagram of a video encoder. An input video contains one or more pictures. A segmentation unit 701 segments a picture in the input video into one or more coding tree units (CTUs). The segmentation unit 701 segments a picture into tiles, and optionally further segments the tiles into one or more bricks, the tiles or bricks containing one or more whole and / or partial CTUs. The segmentation unit 701 forms one or more slices, which may contain one or more tiles in a picture arranged in a raster order of tiles, or one or more tiles covering a rectangular area in a picture. The segmentation unit 701 may also form one or more sub-pictures, which may contain one or more slices, tiles, or bricks.
[0062] In the encoding process of the encoder 700, the partition unit 701 passes the CTU to the prediction unit 702. Generally, the prediction unit 702 is composed of a block partition unit 703, an ME (motion estimation) unit 704, an MC (motion compensation) unit 705, and an intra prediction unit 706. The block partition unit 703 further uses quadtree partitioning, binary partitioning, and ternary partitioning to iteratively partition the input CTU into smaller coding units (CUs). The prediction unit 702 can use the ME unit 704 and the MC unit 705 to derive the inter-frame prediction block of the CU. The intra-frame prediction unit 706 can use various intra-frame prediction modes to derive the intra-frame prediction block of the CU, and the various intra-frame prediction modes include angle prediction mode, DC mode, plane mode, matrix-based intra-frame prediction mode, etc. In an example, the rate-distortion optimization motion estimation method can be called by the ME unit 704 and the MC unit 705 to derive the inter-frame prediction block, and the rate-distortion optimization mode decision method can be called by the intra-frame prediction unit 706 to obtain the intra-frame prediction block.
[0063] The prediction unit 702 outputs the prediction block of the CU. The adder 707 calculates the difference between the CU in the output of the segmentation unit 701 and the prediction block of the CU, that is, the residual CU. The transform unit 708 reads the residual CU and performs one or more transform operations on the residual CU to obtain coefficients. The quantization unit 709 quantizes the coefficients and outputs the quantized coefficients, that is, levels.
[0064] The inverse quantization unit 710 performs a scaling operation on the quantized coefficients to output the reconstructed coefficients. The inverse transform unit 711 performs one or more inverse transforms corresponding to the transform in the transform unit 708 and outputs the reconstructed residual. The adder 712 calculates the reconstructed CU by adding the reconstructed residual to the prediction block of the CU from the prediction unit 702. The adder 712 also forwards its output to the prediction unit 702 for use as an intra-frame prediction reference. After all CUs in the picture or sub-picture are reconstructed, the filtering unit 712 performs loop filtering on the reconstructed picture or sub-picture. The filtering unit 712 includes one or more filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luminance mapping and chrominance scaling (LMCS) filter, and a filter based on a neural network. The filtering unit 712 also includes a CNN-based filter discussed herein. As an example, the filters in the filtering unit 712 can be connected to each other in a cascade order. In addition, when the filtering unit 712 determines that the CU is not used as a reference for encoding other CUs, the filtering unit 712 performs loop filtering on one or more target pixels in the CU.
[0065] The output of the filtering unit 712 is a decoded picture or sub-picture, which is forwarded to the DPB (decoded picture buffer) 713. The DPB 713 outputs the decoded picture according to the timing and control information. The picture stored in the DPB 713 can also be used as a reference for inter-frame or intra-frame prediction performed by the prediction unit 702. Optionally, the decoded picture can be further processed using the CNN-based filter discussed herein to enhance the quality of the decoded picture.
[0066] The entropy coding unit 715 converts the parameters necessary for deriving the decoded image, control parameters and supplementary information in the units of the encoder 700 into binary representations, and writes these binary representations into the generated video bitstream according to the syntax structure of each data unit.
[0067] The encoder 700 may be a computing device including a processor and a storage medium recording an encoding program. When the processor reads and executes the encoding program, the encoder 700 reads the input video and generates a corresponding video bitstream.
[0068] The encoder 700 may be a computing device including one or more chips. Figure 7 The corresponding units in the system have similar functions, similar connections and data exchange.
[0069] Please note that the video encoder 700 may also be used to encode an image with only the block partition unit 703 and the intra prediction unit 706 enabled in the prediction unit 702 .
[0070] Figure 8800 is an exemplary block diagram of a video decoder. The input bitstream of the decoder 800 may be a bitstream generated by the encoder 700. The parsing unit 801 parses the input bitstream and obtains the value of the syntax element from the input bitstream. The parsing unit 801 converts the binary representation of the syntax element into a numerical value and forwards the numerical value to a unit in the decoder 800 to derive one or more decoded pictures. The parsing unit 801 may also parse one or more syntax elements from the input bitstream for displaying the decoded picture.
[0071] The parsing unit 801 forwards the value of the syntax element and one or more variables set or determined according to the value of the syntax element to the units in the decoder 800 to derive one or more decoded pictures. The prediction unit 802 determines the prediction block of the current decoded block (e.g., CU). When the inter-coding mode is indicated for decoding the current decoded block, the prediction unit 802 passes the relevant parameters from the parsing unit 801 to the MC unit 803 to derive the inter-prediction block. When the intra-prediction mode is indicated for decoding the current decoded block, the prediction unit 802 passes the relevant parameters from the parsing unit 801 to the intra-prediction unit 804 to derive the intra-prediction block.
[0072] The scaling unit 805 has the same function as the inverse quantization unit 710 in the encoder 700. The scaling unit 805 performs a scaling operation on the quantized coefficients (ie, levels) from the parsing unit 801 to obtain reconstructed coefficients.
[0073] The transform unit 806 has the same function as the inverse transform unit 711 in the encoder 700. The transform unit 806 performs one or more transform operations (ie, the inverse operations of the one or more transform operations performed by the inverse transform unit 711 in the encoder 700) to obtain a reconstructed residual.
[0074] The adder 807 performs an addition operation on the input of the prediction block from the prediction unit 802 and the reconstructed residual from 806 to obtain a reconstructed block of the current decoded block. The reconstructed block is also sent to the prediction unit 802 to be used as a reference for other blocks encoded in intra prediction mode.
[0075] After all CUs in the picture or sub-picture are reconstructed, the filtering unit 808 performs loop filtering on the reconstructed picture or sub-picture. The filtering unit 808 includes one or more filters, for example, a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luminance mapping and chroma scaling (LMCS) filter, and a filter based on a neural network. The filtering unit 808 also includes a CNN-based filter discussed herein. As an example, the filters in the filtering unit 808 can be connected to each other in a cascade order. In addition, when the filtering unit 808 determines that the reconstructed block is not used as a reference for decoding other blocks, the filtering unit 808 performs loop filtering on one or more target pixels in the reconstructed block.
[0076] The output of the filtering unit 808 is a decoded picture or sub-picture, which is forwarded to the DPB (decoded picture buffer) 809. The DPB 809 outputs the decoded picture according to the timing and control information. The picture stored in the DPB 809 can also be used as a reference for performing inter-frame or intra-frame prediction by the prediction unit 802. Optionally, the decoded picture output from the DPB 809 by the decoder 800 can be further processed using the CNN-based filter discussed herein to enhance the quality of the decoded picture.
[0077] The decoder 800 may be a computing device including a processor and a storage medium recording a decoding program. When the processor reads and executes the decoding program, the decoder 800 reads an input video bitstream and generates a corresponding decoded video.
[0078] The encoder 800 may be a computing device including one or more chips. Figure 8 The corresponding units in the system have similar functions, similar connections and data exchange.
[0079] Note that the video decoder 800 can also be used to decode a bitstream of an image. One example implementation is to enable the intra prediction unit 804 only in the prediction unit 802 of the decoder 800.
[0080] Other considerations
[0081] The above detailed description of the examples of the disclosed technology is not intended to be exhaustive, nor is it intended to limit the disclosed technology to the above precise form. Although the specific examples of the disclosed technology are described above for illustrative purposes, those skilled in the art will recognize that various equivalent modifications can be made within the scope of the technology. For example, although the process or block is presented in a given order, alternative implementations can perform routines with steps in different orders, or adopt a system with blocks, and can delete, move, add, subdivide, combine and / or modify certain processes or blocks to provide alternative implementations or sub-combinations. Each of these processes or blocks can be implemented in a variety of different ways. In addition, although the process or block is sometimes shown as being executed in series, these processes or blocks can be executed or implemented in parallel, or can be executed at different times. In addition, any specific numbers mentioned herein are examples only; alternative implementations can adopt different values or ranges.
[0082] In the detailed description, many specific details are described to provide a thorough understanding of the technology currently described. In other embodiments, the technology introduced herein can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring the present disclosure, well-known features, such as specific functions or routines, are not described in detail. References to "implementation / embodiment", "one implementation / embodiment", etc. in this specification mean that the specific features, structures, materials or characteristics described are included in at least one embodiment of the technology described. Therefore, the appearance of such phrases in this specification does not necessarily refer to the same implementation / embodiment. On the other hand, these references are not necessarily mutually exclusive. In addition, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more implementations / embodiments. It should be understood that the various embodiments shown in the drawings are merely illustrative representations and are not necessarily drawn to scale.
[0083] For the sake of clarity, many details describing structures or processes that are well known and often associated with communication systems and subsystems are not set forth herein, as such details may unnecessarily obscure some important aspects of the disclosed technology. In addition, although the following disclosure sets forth several embodiments of different aspects of the disclosure, other embodiments may have different configurations or different components than those described in this section. Thus, the disclosed technology may have other embodiments that include additional elements or do not include some of the elements described below.
[0084] Many implementations or aspects of the technology described herein can take the form of computer or processor executable instructions, including routines performed by a programmable computer or processor. It will be appreciated by those skilled in the relevant art that the described technology can be practiced on a computer or processor system other than those shown and described below. The technology described herein can be implemented in a special-purpose computer or data processor, which is specially programmed, configured or constructed to perform one or more of the computer executable instructions described below. Therefore, the terms "computer" and "processor" commonly used herein refer to any data processor. The information processed by these computers and processors can be presented on any suitable display medium. Instructions for performing computer or processor executable tasks can be stored or located in any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive and / or other suitable media.
[0085] The term "and / or" in this specification is only used to describe the association relationship between related objects, and indicates that there may be three relationships. For example, A and / or B can represent the following three situations: A exists alone, A and B both exist, and B exists alone.
[0086] In light of the above detailed description, these and other changes may be made to the disclosed technology. Although certain examples of the disclosed technology and the expected best mode are described in detail, no matter how detailed the above description appears in the text, the disclosed technology can be practiced in a variety of ways. Although the details of the system may vary greatly in its specific implementation, they still belong to the scope of the technology disclosed herein. As described above, specific terms used in describing certain features or aspects of the disclosed technology should not be considered to imply that the term is redefined herein to be limited to any specific characteristics, features or aspects of the disclosed technology associated with the term. Therefore, the present invention is not limited except to the appended claims. Generally, the terms used in the appended claims should not be interpreted as limiting the disclosed technology to the specific examples disclosed in the specification unless the detailed description section above explicitly defines these terms.
[0087] Those skilled in the art will appreciate that, in conjunction with the examples described in the embodiments disclosed in this specification, the units and algorithm steps may be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the design constraints of the specific application and technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but this should not be considered beyond the scope of this application.
[0088] Although certain aspects of the invention are presented below in certain claim forms, applicants contemplate the various aspects of the invention in any number of claim forms. Therefore, applicants reserve the right to pursue additional claims after filing this application, to pursue such additional claim forms in this application or in a subsequent application.
Claims
1. A video processing method, characterized in that: The method comprises: receiving an input image; Decomposing the input image into a set of wavelet subbands using discrete wavelet transform; The set of wavelet subbands is input into a neural network framework, The neural network framework includes two subnetworks, the first subnetwork is configured to restore the set of wavelet subbands from high frequency to low frequency, and use the restored high frequency wavelet subbands to restore the low frequency wavelet subbands, and the second subnetwork is configured to use hybrid convolution to expand the size of the receptive field of the restored set of wavelet subbands; and An enhanced version of the input image output from the second sub-network is received from the neural network framework.
2. The method according to claim 1, characterized in that The hybrid convolution includes a combination of dilated convolution and standard convolution.
3. The method according to claim 1, characterized in that The first sub-network also includes one or more Res2Net groups, wherein each Res2Net group includes one or more Res2Net modules.
4. The method according to claim 3, characterized in that Each Res2Net group also includes a first 3×3 convolutional layer at the beginning of the Res2Net group and a second 3×3 convolutional layer at the end of the Res2Net group.
5. The method according to claim 3, characterized in that: Each of the Res2Net modules includes one or more 3×3 convolutional layers between a first 1×1 convolutional layer and a second 1×1 convolutional layer.
6. The method according to claim 5, characterized in that Each of the Res2Net modules includes a spatial attention module and / or a channel attention module after the second 1×1 convolutional layer.
7. The method according to claim 1, characterized in that The second sub-network includes one or more hybrid convolution groups between the first 3×3 convolutional layer and the second 3×3 convolutional layer.
8. The method according to claim 7, characterized in that Each of the mixed convolution groups includes one or more densely connected mixed convolution blocks.
9. The method according to claim 8, characterized in that Each of the mixed convolution groups includes four mixed convolution blocks, and the void coefficients of the four mixed convolution blocks are 1, 2, 4 and 8 respectively.
10. The method according to claim 1, characterized in that The neural network framework applies an inverse discrete wavelet transform to the recovered set of wavelet subbands between the first subnetwork and the second subnetwork.
11. A video processing system, characterized in that: The system comprises: Processor; and A memory configured to store instructions that, when executed by the processor, are used to: receiving an input image; Decomposing the input image into a set of wavelet subbands using discrete wavelet transform; The set of wavelet subbands is input into a neural network framework, The neural network framework includes two subnetworks, the first subnetwork is configured to restore the set of wavelet subbands from high frequency to low frequency, and use the restored high frequency wavelet subbands to restore the low frequency wavelet subbands, and the second subnetwork is configured to use hybrid convolution to expand the size of the receptive field of the restored set of wavelet subbands; and An enhanced version of the input image output from the second sub-network is received from the neural network framework.
12. The system according to claim 11, characterized in that The hybrid convolution includes a combination of dilated convolution and standard convolution.
13. The system according to claim 11, characterized in that The first sub-network also includes one or more Res2Net groups, wherein each of the Res2Net groups includes one or more Res2Net modules.
14. The system according to claim 13, characterized in that Each of the Res2Net groups further includes a first 3×3 convolutional layer at the beginning of the Res2Net group and a second 3×3 convolutional layer at the end of the Res2Net group.
15. The system according to claim 13, characterized in that Each of the Res2Net modules includes one or more 3×3 convolutional layers between a first 1×1 convolutional layer and a second 1×1 convolutional layer.
16. The system according to claim 15, characterized in that Each of the Res2Net modules includes a spatial attention and a channel attention module after the second 1×1 convolutional layer.
17. The system according to claim 11, characterized in that The second sub-network includes one or more hybrid convolution groups between the first 3×3 convolutional layer and the second 3×3 convolutional layer.
18. The system according to claim 11, characterized in that Each of the mixed convolution groups includes one or more densely connected mixed convolution blocks.
19. The system according to claim 18, characterized in that Each of the mixed convolution groups includes four mixed convolution blocks, and the void coefficients of the four mixed convolution blocks are 1, 2, 4 and 8 respectively.
20. A video processing method, characterized in that: The method comprises: receiving an input image; applying a discrete wavelet transform to the input image to form a set of wavelet subbands; inputting the set of wavelet subbands into a first network, the first network being configured to recover the set of wavelet subbands from high frequency to low frequency; applying an inverse discrete wavelet transform to the recovered wavelet subbands to create a reconstructed signal; Inputting the reconstructed signal into a second network, the second network being configured to expand the size of the receptive field using hybrid convolution; and An enhanced version of the input image is received from the second network.
