Partition information guided reference picture resampling (rpr)-based super-resolution

Abstract

Methods and systems for video processing are provided. In some embodiments, the method comprises: (i) receiving an input image; (ii) processing the input image by one or more convolutional layers; (iii) processing the input image by the plurality of residual blocks by using the division information of the input image as a reference to obtain a reference information feature; (iv) generating features of different scales based on the reference information features; (v) processing the features of different scales by a plurality of convolutional layer sets; (vi) processing the features of different scales by a reference spatial attention block (RSAB) to form a combined feature; and (vii) concatenating the combined feature with the reference information feature to form an output image.

Description

Technical Field

[0001] The present disclosure relates to image processing. For example, the present disclosure includes a video compression scheme that can improve video reconstruction performance and efficiency. More specifically, the present disclosure relates to systems and methods for providing convolutional neural network filters for upsampling processes. Background Art

[0002] Video coding of high-definition video has been the focus of attention over the past decade. Although coding techniques have improved, it remains challenging to transmit high-definition video over limited bandwidth. Approaches to this problem include resampling-based video coding, in which (i) before encoding, the original video is first "downsampled" to form an encoded video at an encoder side (the encoder includes a decoder for generating a bitstream), (ii) the encoded video is transmitted to a decoder side as a bitstream, and then the bitstream is decoded in a decoder (the decoder is the same as the decoder included in the encoder) to form a decoded video; and (iii) the decoded video is then "upsampled" to the same resolution as the original video. For example, Versatile Video Coding (VVC) supports a resampling-based coding scheme (reference picture resampling, RPR) that enables temporal prediction between different resolutions. However, conventional methods cannot efficiently handle the upsampling process, especially for videos with complex characteristics. Therefore, it is advantageous to have an improved system and method to address the aforementioned needs. Summary of the invention

[0003] The present disclosure relates to systems and methods for improving the image quality of a video using a neural network for video compression. More specifically, the present disclosure provides attention-based super-resolution (SR) guided by partitioning information for video compression. In some embodiments, a convolutional neural network (CNN) is combined with an RPR function in a VVC to achieve super-resolution reconstruction (e.g., removing artifacts). More specifically, the present disclosure utilizes reconstructed frames and upsampled frames obtained by the RPR function as input, and then uses coding tree unit (CTU) partitioning information (e.g., a CTU partition map) as a reference to generate spatial attention information for removing artifacts.

[0004] In some embodiments, considering the correlation between the luminance component and the chrominance component, features are extracted through three branches for the luminance component and the chrominance component. The extracted features are then cascaded and fed to the "U-net" structure. The SR reconstruction result is then generated through three reconstruction branches.

[0005] In some embodiments, the "U-Net" structure includes multiple stacked attention blocks (e.g., dense blocks based on dilated convolutional layers with channel attention, DDBCA). The "U-Net" structure is configured to efficiently extract low-level features and then transmit the extracted low-level features to a high-level feature extraction module (e.g., by skipping connections in the U-Net structure). High-level features contain global semantic information, while low-level features contain local detail information. U-Net connections can further reuse low-level features while recovering local details.

[0006] One aspect of the present disclosure is to use only the segmentation information as a reference when processing images / videos (e.g., see Figure 2 ), rather than as an input. With this arrangement, the present disclosure can effectively combine features affected by the segmentation information without excessively increasing the undesirable negative effects caused by directly inputting the segmentation information into the image / video.

[0007] Another aspect of the present disclosure is to process the luma component and the chroma component simultaneously while using the partition information as a reference. Figure 2 ), the present disclosure provides a framework or network that can process luma components and chroma components simultaneously while paying attention to the split information.

[0008] Another aspect of the present disclosure is to provide an efficient encoding strategy based on resampling. The system and method can effectively reduce the transmission bandwidth to avoid or mitigate the degradation of video quality.

[0009] In some embodiments, the method may be implemented by a tangible, non-transitory computer-readable medium having stored thereon processor instructions that, when executed by one or more processors, cause the one or more processors to perform one or more aspects / features of the method described herein. In other embodiments, the method may be implemented by a system including a computer processor and a non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor, cause the computer processor to perform one or more actions of the method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following briefly describes the drawings. The drawings only show some aspects or embodiments of the present disclosure, and ordinary technicians in this field can still derive other drawings from these drawings without creative efforts.

[0011] Figure 1 is a schematic diagram illustrating an upsampling process in resampling-based video encoding according to one or more embodiments of the present disclosure.

[0012] Figure 2 is a schematic diagram illustrating an RPR-based super-resolution (SR) framework (ie, CNN filters in the upsampling process) according to one or more embodiments of the present disclosure.

[0013] Figure 3 is a schematic diagram illustrating a reference spatial attention block (RSAB) according to one or more embodiments of the present disclosure.

[0014] Figure 4 is a schematic diagram illustrating a dilated convolutional layer based dense block with channel attention (DDBCA) according to one or more embodiments of the present disclosure.

[0015] 5a-5e are images illustrating test results according to one or more embodiments of the present disclosure.

[0016] Figure 6 and Figure 7 is a test result of a framework according to one or more embodiments of the present disclosure.

[0017] Figure 8 is a schematic diagram of a wireless communication system according to one or more embodiments of the present disclosure.

[0018] Fig. 9 is a schematic block diagram of a terminal device according to one or more embodiments of the present disclosure.

[0019] Fig.10 is a schematic block diagram of a device according to one or more embodiments of the present disclosure.

[0020] Fig.11 is a flow chart of a method according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following briefly describes the drawings. The drawings only show some aspects or embodiments of the present disclosure, and ordinary technicians in this field can still derive other drawings from these drawings without creative efforts.

[0022] Figure 1is a schematic diagram illustrating an upsampling process in a resampling-based video coding 100 according to one or more embodiments of the present disclosure. In order to implement the RPR function in the resampling-based video coding, the current frame for encoding is first downsampled to reduce the bitstream transmission, and then the frame is restored at the decoding end. The current frame will be upsampled to its original resolution. The upsampling process 100 includes an SR neural network to replace the traditional upsampling algorithm in the traditional RPR configuration. The upsampling process 100 may include a CNN filter 101 having a dense block based on an expanded convolutional layer with an attention mechanism. The upsampling process 100 uses residual learning to reduce the complexity of network learning, thereby improving performance and efficiency.

[0023] like Figure 1 As shown, the image can be sent from the loop filter 103 for the upsampling process 10. In some embodiments, the loop filter 103 can be applied in the encoding and decoding loop after the inverse quantization process and before the processed image is stored in the decoded picture buffer 105. In the upsampling process 10, the RPR upsampling module 107 receives the image 11 from the loop filter 103, and then generates an upsampled frame 12 and transmits it to the CNN filter 101. The loop filter 103 also sends the reconstructed frame 11 to the CNN filter 101. The CNN filter 101 then processes the upsampled frame 12 and the reconstructed frame 11 and sends the processed image 16 to the decoded picture buffer 105 for further processing (e.g., generating a decoded video sequence).

[0024] Figure 2 2 is a schematic diagram illustrating a framework 200 of RPR-based SR guided by partition information. Figure 2 As shown, the framework 200 includes four parts: a feature extraction part 201, a reference information generation (RIG) part 203, a mutual information processing part 205, and a reconstruction part 207. When processing a video / image, the framework 200 uses the segmentation information 222 as a reference (rather than as an input). As described in detail below, the segmentation information 222 is used in the RIG part 203 (e.g., via the residual block 2031) and the mutual information processing part 205 (e.g., via the reference feature attention module 2052). Please note that for ease of reference, these parts are described separately, and these parts can work together during processing.

[0025] The feature extraction part 201 includes three convolutional layers (201a-c). The convolutional layers 201a-c are used to extract features of the input 21 (e.g., the luminance component "Y" and the chrominance components "Cb" and "Cr"). The convolutional layers 201a-c are followed by a ReLU (rectified linear unit) activation function. In some embodiments, the input may be a reconstructed frame after the RPR upsampling process. In some embodiments, the input may include a luminance component and / or a chrominance component.

[0026] In some embodiments, assuming that the input Y Rec , and Through the feature extraction layers "cy1", "cb1" and "cr1", the extracted features and It can be expressed as follows.

[0027]

[0028] The reference information generation (RIG) section 203 includes eight residual blocks 2031 (in Figure 2 1, 2, 3, 4, 5, 6, 7, and 8 in FIG. 2 ). The first four residual blocks 2031 (e.g., numbers 1, 2, 3, and 4) are executed to predict CTU partition information from the reconstructed frame of the input 21. A reference residual block (e.g., number 5) is generated and used to merge the partition information 222. The following three residual blocks 2031 (e.g., numbers 6, 7, and 8) are used for reference information generation.

[0029] Subsequently, the reference information features can be used as input to several convolutional layer sets 2032 to generate features of different scales, which can be used as input to a reference feature attention module (e.g., reference spatial attention block 2052, described below). Each convolutional layer set 2032 may include a convolutional layer with a stride of 2 (in Figure 2 2032a) and a convolutional layer followed by a ReLU (in Figure 2 Therefore, the output of the RIG part 203 can be expressed as follows:

[0030]

[0031] The mutual information processing (MIP) part 205 is based on the U-Net backbone. The input of the MIP part 205 can be the reference feature f r as well as and cascade.

[0032] The MIP part 205 includes a convolutional layer 2051, a reference spatial attention block (RSAB) 2052, and a dense block based on dilated convolutional layers with channel attention (DDBCA) 2053.

[0033] like Figure 2 As shown, there are four different scales 205A-D in the MIP part 205 (e.g., four horizontal branches below the RIG part 203). The first three scales (e.g., starting from the top, 205A-C) use two DDBCA 2053 followed by one RSAB 2052, while the last scale (e.g., at the bottom, 205D) uses four DDBCA 2053 followed by one RSAB 2052. Finally, the combined feature f is generated by reconstructing the multi-scale features as follows c :

[0034]

[0035] The reconstruction section 207 includes three branch paths for processing the luminance component and the chrominance component. In some embodiments, for the luminance channel (path 2071), the combined feature f c It is upsampled and put into three convolutional layers 2071a, followed by addition 2071b with the reconstructed luminance component 209 after the RPR upsampling process.

[0036] In some embodiments, for the chrominance channel (e.g., paths 2072, 2073), the combined feature f c With the extracted features and Cascaded, and then input to three convolutional layers 2072a, 2073a. The final output is generated as follows:

[0037]

[0038] Figure 3 300 is a schematic diagram illustrating a reference spatial attention block (RSAB) 300 according to one or more embodiments of the present disclosure. The block artifacts shown in the decoding are closely related to the block partitioning. Therefore, the CTU partition map is suitable as auxiliary information for predicting block artifacts. When the partition map is used directly as input, the block artifacts of the partition map will have a negative impact on super-resolution. Therefore, the present disclosure uses RSAB 300 to guide the image deblocking process by analyzing the CTU partition information in the CTU partition map.

[0039] like Figure 3 As shown, RSAB 300 includes three convolutional layers 301a-c followed by a ReLU function 303 and a Sigmoid function 305. Figure 2Those discussed) are sequentially put into convolutional layers 301a-c, ReLU function 303 and Sigmoid function 305. Finally, the input features are multiplied (e.g., at 307) by the processed reference features. The "dashed line" ( Figure 3 The upper part of the ) shows the main processing flow ( Figure 3 Compared to the solid line in the lower part of the figure), the partition information is used only as a reference and not as an input.

[0040] In order to reduce the number of parameters and expand the receptive field of the image, the present disclosure integrates the dilated convolutional layer and the channel attention module into a “dense block”, e.g. Figure 4 shown. Figure 4 4 is a schematic diagram illustrating a dilated convolutional layer based dense block with channel attention (DDBCA) 400 according to one or more embodiments of the present disclosure. The DDBCA 400 includes a dilated convolution based dense module 401 and an optimized channel attention module 403.

[0041] In some embodiments, the dilated convolution based dense module 401 includes one convolution layer 4011 and three dilated convolution layers 4012. The three dilated convolution layers 4012 include layers 4012a (with dilation factor 2), 4012b (with dilation factor 2), and 4012c (with dilation factor 4). With this arrangement, the receptive field of the dilated convolution based dense module 401 is larger than the receptive field of a normal convolution layer.

[0042] In some embodiments, the optimized channel attention module 403 is configured to perform a squeeze and excitation (SE) attention mechanism, so it can be referred to as an SE attention module. Compared with the ordinary channel attention module, the optimized channel attention module 403 is configured to enhance the nonlinear relationship between the input feature channels. The optimized channel attention module 403 is configured to perform three steps, including a "squeeze" step, an "excitation" step, and a "scaling" step.

[0043] Extrusion step (4031): First, global average pooling is performed on the input feature map to obtain f sq Each learned filter operates with a local receptive field, so each unit of the transformed output cannot exploit contextual information outside of that region. To alleviate this problem, the SE attention mechanism first “squeezes” the global spatial information into the channel descriptor. This is achieved by generating channel-wise statistics through global average pooling.

[0044] Stimulus Step (4033):This step aims to better obtain the dependencies of each channel. Two conditions need to be met: the first condition is that the nonlinear relationship between each channel can be learned, and the second condition is that each channel has an output (for example, the value cannot be 0). The activation function in the embodiment shown can be "sigmoid" instead of the commonly used ReLU. The excitation process is f sq After two fully connected layers, channels are compressed and restored. In image processing, in order to avoid conversion between matrices and vectors, 1×1 convolutional layers are used instead of fully connected layers.

[0045] Zoom: Finally, a dot product is performed between the output after excitation and the SE attention. With this arrangement, the intrinsic relationship of the features using the adaptive channel weight map can be established.

[0046] In some embodiments, L1 or L2 loss can be used to train the proposed framework discussed in this article. The loss function f(x) can be expressed as follows:

[0047]

[0048] Where "α" is a coefficient that balances L1 and L2 losses, "epochs" is the total number of epochs in the training process, and "epoch" is the current index. At the beginning of training, L1 loss has a larger weight to speed up convergence, while in the second half of training, L2 loss plays an important role in generating better results. In some embodiments, L1 or L2 loss is a loss function that compares at the pixel level. L1 loss calculates the sum of the absolute values ​​of the differences between the output and the true data, while L2 loss calculates the sum of the squares of the differences between the output and the true data.

[0049] Figures 5a to 5e (i.e., "Cat Robot") are images illustrating test results according to one or more embodiments of the present disclosure. The images are described as follows: (a) original image; (b) image processed according to an existing standard (VTM 11.0NNVC-1.0, denoted as "anchor"); (c) a portion of the original image to be compared; (d) image processed with the RPR process; and (e) image processed by the framework discussed in this article. It can be seen and supported by the test results below that the present framework (i.e., (e)) effectively provides better image quality compared to existing methods (i.e., (b) and (d)).

[0050] Table 1 below shows quantitative measurements of the use of this framework. Test results under the "All Intra-frame" (AI) configuration. Among them, "bold numbers" represent positive gains, while "underlined" numbers represent negative gains. These tests were all performed under "CTC". "VTM-11.0" and the new "MCTF" were used as baselines for testing. Table 1 shows the comparison results with the VTM 11.0NNVC-1.0 anchor. This framework achieved a BD rate reduction of {-9.25%, 8.82%, -16.39%} under the AI ​​configuration.

[0051]

[0052] Figure 6 and Figure 7 is a test result of a framework according to one or more embodiments of the present disclosure. Figure 6 and Figure 7 The rate distortion (RD) curve is used to demonstrate the test results. "A" represents the average value of different groups (A1 and A2). The RD curves of A1 sequence and A2 sequence are shown in Figure 6 and Figure 7 As shown in the figure, the proposed framework (labeled as "proposed") has achieved significant results in all A1 sequences and A2 sequences. Among them, all RD curves of the proposed framework exceed the RD curve of VTM-11.0 in the lower bit rate region (i.e., the left part of the curve), which shows that the proposed framework is more efficient at low bandwidth.

[0053] Figure 8 8 is a schematic diagram of a wireless communication system 800 according to one or more embodiments of the present disclosure. The wireless communication system 800 may implement the framework discussed herein. Figure 8As shown, the wireless communication system 800 may include a network device (or base station) 801. Examples of the network device 801 include a base transceiver station (Base Transceiver Station, BTS), a node B (NodeB, NB), an evolved node B (eNB or eNodeB), a next-generation node B (gNB or gNode B), a wireless fidelity (Wi-Fi) access point (AP), etc. In some embodiments, the network device 801 may include a relay station, an access point, a vehicle-mounted device, a wearable device, etc. The network device 801 may include a wireless connection device for the following communication networks: a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a wideband CDMA (WCDMA) network, an LTE network, a cloud radio access network (Cloud Radio Access Network, CRAN), a network based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 (e.g., a Wi-Fi network), an Internet of Things (IoT) network, a device-to-device (D2D) network, a next-generation network (e.g., a 5G network), a future-evolved public land mobile network (Public Land Mobile Network, PLMN), etc. The 5G system or network may be referred to as a New Radio (NR) system or network.

[0054] exist Figure 8 , the wireless communication system 800 also includes a terminal device 803. The terminal device 803 may be an end user device configured to facilitate wireless communication. The terminal device 803 may be configured to be wirelessly connected to the network device 801 (e.g., via a wireless channel 805) according to one or more corresponding communication protocols / standards. The terminal device 803 may be mobile or fixed. The terminal device 803 may be a user equipment (UE), an access terminal, a user unit, a user station, a mobile site, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent, or a user device. Examples of the terminal device 803 include a modem, a cellular phone, a smart phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or another processing device connected to a wireless modem, a vehicle-mounted device, a wearable device, an Internet of Things (IoT) device, a device used in a 5G network, a device used in a public land mobile network, etc. For the purpose of illustration, Figure 8 Only one network device 801 and one terminal device 803 are illustrated in the wireless communication system 800. However, in some examples, the wireless communication system 800 may include additional network devices 801 and / or terminal devices 803.

[0055] Fig. 9 It is a schematic block diagram of a terminal device 903 (for example, which can implement the method discussed herein) according to one or more embodiments of the present disclosure. As shown in the figure, the terminal device 903 includes a processing unit 910 (for example, a DSP, a CPU, a GPU, etc.) and a memory 920. The processing unit 910 can be configured to implement instructions corresponding to the methods discussed herein and / or other aspects of the above-mentioned embodiments. It should be understood that the processor 910 in the embodiment of the present technology can be an integrated circuit chip and has signal processing capabilities. During implementation, the steps in the above method can be implemented by using an integrated logic circuit of hardware in the processor 910 or using instructions in the form of software. The processor 910 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or another programmable logic device, a discrete gate or transistor logic device, and a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present technology can be implemented or executed. The general-purpose processor 910 can be a microprocessor, or the processor 910 can be any conventional processor, etc. instead. The steps in the method disclosed with reference to the embodiments of the present technology can be directly executed or completed by a decoding processor implemented as hardware, or by using a combination of hardware modules and software modules in the decoding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or another mature storage medium in the art. The storage medium is located in the memory 920, and the processor 910 reads the information in the memory 920 and completes the steps in the above method in combination with its hardware.

[0056] It is understood that the memory 920 in the embodiment of the present technology can be a volatile memory or a non-volatile memory, or can include both a volatile memory and a non-volatile memory. Non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. Volatile memory can be a random access memory (RAM) and used as an external cache. For exemplary and non-restrictive descriptions, various forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct Rambus random access memory (DR RAM). It should be noted that the memory in the system and method described herein is intended to include but is not limited to these memories and any other suitable type of memory. In some embodiments, the memory can be a non-transient computer-readable storage medium storing instructions that can be executed by a processor.

[0057] Fig.10 1 is a schematic block diagram of a device 1000 according to one or more embodiments of the present disclosure. The device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0058] The processing component 1002 generally controls the overall operation of the electronic device, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps in the above method. In addition, the processing component 1002 may include one or more modules that facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 may include a multimedia module to facilitate interaction between the multimedia component 1008 and the processing component 1002.

[0059] The memory 1004 is configured to store various types of data to support the operation of the electronic device. Examples of such data include instructions for any application or method running on the electronic device, contact data, phone book data, messages, pictures, videos, etc. The memory 1004 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, and magnetic or optical disks.

[0060] The power supply component 1006 provides power to various components of the electronic device. The power supply component 1006 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device.

[0061] Multimedia component 1008 may include a screen providing an output interface between an electronic device and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen may include a TP, the screen may be implemented as a touch screen to receive input signals from a user. The TP may include one or more touch sensors to sense touch, slide, and gestures on the TP. The touch sensor may not only sense the boundaries of a touch or slide action, but may also detect the duration and pressure associated with a touch or slide action. In some embodiments, multimedia component 1008 may include a front camera and / or a rear camera. When the electronic device is in an operating mode such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system, or may have focusing and optical zoom capabilities.

[0062] The audio component 1010 is configured to output and / or input audio signals. For example, the audio component 1010 may include a microphone (MIC), and the MIC is configured to receive external audio signals when the electronic device is in an operation mode such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may be further stored in the memory 1004 or sent via the communication component 1016. In some embodiments, the audio component 1010 may further include a speaker configured to output an audio signal.

[0063] The I / O interface 1012 provides an interface between the processing component 1002 and a peripheral interface module, and the peripheral interface module may be a keyboard, a click wheel, a button, etc. The button may include but is not limited to a home button, a volume button, a start button, and a lock button.

[0064] The sensor component 1014 may include one or more sensors configured to provide state assessment for the electronic device in various aspects. For example, the sensor component 1014 can detect the on / off state of the electronic device and the relative positioning of components such as the display and keypad of the electronic device, and the sensor component 1014 can further detect the position change of the electronic device or the component of the electronic device, whether there is contact between the user and the electronic device, the orientation or acceleration / deceleration of the electronic device, and the temperature change of the electronic device. The sensor component 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 1014 may also include a light sensor configured for imaging applications, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor. In some embodiments, the sensor component 1014 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0065] Communication component 1016 is configured to promote wired or wireless communication between electronic device and other devices. Electronic device can access wireless network based on communication standard, such as WIFI network, second generation (2G) or 3G network or its combination. In an exemplary embodiment, communication component 1016 receives broadcast signal or broadcast associated information from external broadcast management system through broadcast channel. In an exemplary embodiment, communication component 1016 can further include near field communication (NFC) module, to promote short-range communication. For example, NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, BT technology and other technologies.

[0066] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and configured to perform the above-described methods.

[0067] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, and the instructions can be executed by a processor 1002 of an electronic device to implement the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0068] Fig.1111 is a flow chart of a method according to one or more embodiments of the present disclosure. The method 1100 may be implemented by a system (such as a system having a framework as discussed herein). The method 1100 is used to enhance image quality (particularly for upsampling processes). The method 1100 includes, at block 1101, receiving an input image.

[0069] At block 1103, method 1100 continues by processing the input image by one or more convolutional layers. In some embodiments, the one or more convolutional layers are part of a feature extraction portion of the framework (e.g., Figure 2 Component 201).

[0070] At block 1105, method 1100 uses partition information of an input image (eg, Figure 2 Component 222) is used as a reference to process the input image to obtain reference information features to continue.

[0071] In some embodiments, the plurality of residual blocks belong to a reference information generation (RIG) portion of the framework. The plurality of residual blocks may include eight residual blocks. In such embodiments, the first four residual blocks may be used to predict coding tree unit (CTU) partition information from one or more convolutional layers.

[0072] At block 1107, the method 1100 continues by generating features of different scales based on the reference information features. At block 1109, the method 1100 continues by processing the features of different scales by a plurality of sets of convolutional layers. At block 1111, the method 1100 continues by processing the features of different scales by a reference spatial attention block (RSAB) to form a combined feature.

[0073] In some embodiments, the method 1100 further includes processing the features of different scales by a dense block based on dilated convolutional layers with channel attention (DDBCA) to form a combined feature. DDBCA and RSAB can belong to the mutual information processing (MIP) part of the framework.

[0074] In some embodiments, the MIP portion includes four scales configured to generate features of different scales. In some embodiments, at least one of the four scales includes two DDBCAs followed by one RSAB. In some embodiments, one of the four scales includes four DDBCAs followed by one RSAB.

[0075] In some embodiments, the RIG part may further include a plurality of convolutional layer sets, and each of the plurality of convolutional layer sets includes a convolutional layer with a stride of 2 and a convolutional layer followed by a rectified linear unit (ReLU).

[0076] At block 1113, method 1100 continues by concatenating the combined features with the reference information features to form an output image. In some embodiments, the combined features are concatenated by a reconstruction portion of the framework. In some embodiments, the reconstruction portion includes three branch paths for processing the luminance component and the chrominance component, respectively.

[0077] Other considerations

[0078] The above specific implementation of the example of the disclosed technology is not intended to be exhaustive or limit the disclosed technology to the precise form disclosed above. Although the specific examples of the disclosed technology are described above for the purpose of illustration, it will be recognized by those skilled in the relevant art that various equivalent modifications are possible within the scope of the described technology. For example, although the process or frame is presented in a given order, the alternative implementation can perform a routine with steps in a different order, or adopt a system with a frame, and some processes or frames can be deleted, moved, added, subdivided, combined and / or modified to provide an alternative implementation or sub-combination. Each of these processes or frames can be implemented in various different ways. In addition, although the process or frame is sometimes shown as being executed in series, these processes or frames can be executed or implemented in parallel instead, or can be executed at different times. In addition, any specific numbers mentioned herein are merely examples; alternative implementations can adopt different values ​​or ranges.

[0079] In the specific embodiments, many specific details are set forth to provide a thorough understanding of the technology currently described. In other embodiments, the technology introduced herein can be practiced without these specific details. In other cases, in order to avoid unnecessarily obscuring the present disclosure, well-known features, such as specific functions or routines, are not described in detail. References to "implementation / embodiment", "one implementation / embodiment", etc. in this specification mean that the specific features, structures, materials or characteristics described are included in at least one embodiment of the technology described. Therefore, these phrases appearing in this specification do not necessarily all refer to the same implementation / embodiment. On the other hand, such references are not necessarily mutually exclusive. In addition, in one or more implementations / embodiments, specific features, structures, materials or characteristics may be combined in any suitable manner. It should be understood that the various implementations shown in the drawings are merely illustrative representations and are not necessarily drawn to scale.

[0080] For the sake of clarity, several details describing structures or processes are not set forth herein, which are well known and commonly associated with communication systems and subsystems, but which may unnecessarily obscure some important aspects of the disclosed technology. In addition, although the following disclosure sets forth several embodiments of different aspects of the present disclosure, several other embodiments may have different configurations or different components than those described in this section. Therefore, the disclosed technology may have other embodiments with additional elements or without several of the elements described below.

[0081] Many embodiments or aspects of the technology described herein can take the form of computer-executable or processor-executable instructions, including routines executed by a programmable computer or processor. It will be appreciated by those skilled in the relevant art that the described technology can be practiced on a computer or processor system other than those shown and described below. The technology described herein can be implemented in a special-purpose computer or data processor, which is specially programmed, configured or constructed to execute one or more computer-executable instructions described below. Therefore, the terms "computer" and "processor" commonly used herein refer to any data processor. The information processed by these computers and processors can be presented on any suitable display medium. Instructions for performing computer-executable or processor-executable tasks can be stored in or on any suitable computer-readable medium, which includes hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive and / or other suitable media.

[0082] The term "and / or" in this specification is only used to describe the association relationship of associated objects and indicates that there may be three relationships. For example, A and / or B can indicate the following three situations: A exists alone, A and B both exist, and B exists alone.

[0083] In view of the above detailed description, these and other changes can be made to the disclosed technology. Although the detailed description describes certain examples of the disclosed technology, as well as the best mode contemplated, no matter how detailed the above description is in words, the disclosed technology can be practiced in many ways. The details of the system may vary greatly in its specific implementation, but still be covered by the technology disclosed herein. As described above, the use of specific terms in describing certain features or aspects of the disclosed technology should not mean that the term is redefined herein to be limited to any specific characteristics, features or aspects of the disclosed technology associated with the term. Therefore, the present invention is not limited except for the attached claims. In general, the terms used in the claims should not be interpreted as limiting the disclosed technology to the specific examples disclosed in the specification unless such terms are explicitly defined in the above-mentioned detailed description section.

[0084] Those skilled in the art will appreciate that, in conjunction with the examples described in the embodiments disclosed in this specification, the units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions of each specific application, but it should not be considered that the implementation is beyond the scope of this application.

[0085] Although certain aspects of the invention are presented below in certain claim forms, applicants contemplate the various aspects of the invention in any number of claim forms. Applicants therefore reserve the right to file additional claims after filing this application, to pursue such additional claim forms in this application or in a continuation application.

Claims

1. A method for image processing, the method comprising: receiving an input image; Processing the input image by one or more convolutional layers; Processing the input image by using the division information of the input image as a reference by a plurality of residual blocks to obtain reference information features; generating features of different scales based on the reference information features; Processing the features of different scales by multiple convolutional layer sets; Processing the features of the different scales by a reference spatial attention block (RSAB) to form a combined feature; and The combined feature is concatenated with the reference information feature to form an output image.

2. The method of claim 1, wherein: The one or more convolutional layers belong to the feature extraction part of the framework.

3. The method of claim 1, wherein: The multiple residual blocks belong to the reference information generation (RIG) part of the framework.

4. The method of claim 3, wherein: The plurality of residual blocks include eight residual blocks, wherein first four residual blocks are used to predict coding tree unit (CTU) partition information from the one or more convolutional layers.

5. The method of claim 4, wherein: The RIG part also includes the multiple convolutional layer sets, wherein each of the multiple convolutional layer sets includes a convolutional layer with a stride of 2 and a convolutional layer followed by a rectified linear unit (ReLU).

6. The method of claim 1, further comprising processing the features of the different scales by a dense block based dilated convolutional layer with channel attention (DDBCA) to form the combined feature.

7. The method of claim 6, wherein: The DDBCA and the RSAB belong to the mutual information processing (MIP) part of the framework.

8. The method of claim 7, wherein: The MIP portion includes four scales configured to generate features at the different scales.

9. The method of claim 8, wherein: At least one of the four scales includes two DDBCAs followed by one RSAB.

10. The method of claim 8, wherein: One of the four scales includes four DDBCAs followed by one RSAB.

11. The method of claim 1, wherein: The combined features are concatenated by the reconstruction part of the framework.

12. The method of claim 11, wherein: The reconstruction part includes three branch paths for processing luminance components and chrominance components respectively.

13. A system for video processing, the system comprising: processor; as well as a memory configured to store instructions that, when executed by the processor, are used to: receiving an input image; Processing the input image by one or more convolutional layers; Processing the input image by using the division information of the input image as a reference by a plurality of residual blocks to obtain reference information features; generating features of different scales based on the reference information features; Processing the features of different scales by multiple convolutional layer sets; Processing the features of the different scales by a reference spatial attention block (RSAB) to form a combined feature; and The combined feature is concatenated with the reference information feature to form an output image.

14. The system of claim 13, wherein: The one or more convolutional layers belong to the feature extraction part of the framework.

15. The system of claim 13, wherein: The multiple residual blocks belong to the reference information generation (RIG) part of the framework.

16. The system of claim 15, wherein: The multiple residual blocks include eight residual blocks, wherein the first four residual blocks are used to predict coding tree unit (CTU) partition information from the one or more convolutional layers, wherein the RIG part further includes the multiple convolutional layer sets, each of the multiple convolutional layer sets including a convolutional layer with a stride of 2 and a convolutional layer followed by a rectified linear unit (ReLU).

17. The system of claim 13, wherein: The features of different scales are processed by a dense block based dilated convolutional layer with channel attention (DDBCA) to form the combined feature.

18. The system of claim 17, wherein: The DDBCA and the RSAB belong to the mutual information processing (MIP) part of the framework.

19. The system of claim 17, wherein: The MIP portion includes four scales configured to generate features at the different scales.

20. A method for video processing, the method comprising: receiving an input image; Processing the input image by one or more convolutional layers; Processing the input image by using the division information of the input image as a reference by a plurality of residual blocks to obtain reference information features; generating features of different scales based on the reference information features; Processing the features of different scales by multiple convolutional layer sets; The features of the different scales are processed by a reference spatial attention block (RSAB) and a dense block based on dilated convolutional layers with channel attention (DDBCA) to form a combined feature; and The combined feature is concatenated with the reference information feature to form an output image.

Images