Video super-resolution processing method and device, storage medium and chip

Abstract

The invention relates to a video super-resolution processing method and device, a storage medium and a chip. The method comprises the steps of obtaining a to-be-processed video frame; performing video super-division on the brightness component of the video frame to be processed through a video super-division model deployed on a neural network accelerator, and performing interpolation processing on the chrominance component of the video frame to be processed; and integrating the brightness component obtained after super-division and the chrominance component obtained after interpolation. The interpolation operation of the U and V channel data and the super-division of the Y channel data are solidified in the video super-division model at the same time, so that the super-division of the Y channel data and the interpolation operation of the U and V channel data can be executed in parallel on the APU, the interpolation of the U and V channel data can be executed in parallel on the APU on the basis that only the calculation amount of the super-division of the Y channel data is reduced, and the calculation amount of the super-division of the Y channel data is reduced. According to the invention, faster video super-component speed and lower power consumption can be obtained, the video is smooth and not blocked, the operation power consumption is low, the heating phenomenon is reduced, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video super-resolution processing method, device, storage medium and chip. Background Art

[0002] In recent years, with the development of embedded hardware platforms, good conditions have been provided for the implementation of deep learning algorithms. In particular, many hardware manufacturers have provided dedicated hardware acceleration engines for artificial intelligence (AI) algorithms, such as Qualcomm's Snapdragon Neural Processing Engine (SNPE) and MediaTek's neuropilot. In this hardware specially customized for AI, the performance of algorithm operation time and power consumption has been improved.

[0003] The task of video super-resolution is to restore the corresponding high-resolution counterpart from a given low-resolution video. When the video super-resolution AI algorithm is deployed on mobile phones and other terminals, with the help of the hardware acceleration engine deployed on the terminal, heterogeneous operations can be performed between the CPU, GPU and accelerated processing unit (APU). According to the characteristics of the work, a clear division of labor is made and different tasks are assigned to the CPU, GPU, and APU, so that they can maximize their respective strengths and achieve the best AI computing performance. At this stage, most video super-resolution algorithms super-resolution RGB three-channel data, or video super-resolution YUV three-channel data, but all three channels are calculated using deep learning algorithms, and the amount of calculation is very large, which is not friendly to terminals with limited computing power. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a video super-resolution processing method, device, storage medium and chip.

[0005] According to a first aspect of an embodiment of the present disclosure, a video super-resolution processing method is provided, comprising:

[0006] Get the video frame to be processed;

[0007] Performing video super-resolution on the luminance component of the video frame to be processed and performing interpolation processing on the chrominance component of the video frame to be processed by using a video super-resolution model deployed on the neural network accelerator;

[0008] The luminance component obtained after super-resolution and the chrominance component obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0009] Optionally, the video super-resolution model includes a video super-resolution network and an interpolation processing unit;

[0010] Wherein, the video super-resolution network is used to perform video super-resolution on the brightness component of the video frame to be processed;

[0011] The interpolation processing unit is used to perform interpolation processing on the chrominance component of the video frame to be processed.

[0012] Optionally, the video super-resolution model is trained in the following manner:

[0013] Acquire a high-resolution video frame sample and a low-resolution video frame sample corresponding to the high-resolution video frame sample;

[0014] The video super-resolution model is obtained by performing model training by taking the luminance component of the low-resolution video frame sample as the input of the video super-resolution network, taking the chrominance component of the low-resolution video frame sample as the input of the interpolation processing unit, and taking the luminance component of the high-resolution video frame sample as the target output of the video super-resolution network.

[0015] Optionally, the acquiring the high-resolution video frame samples and the low-resolution video frame samples corresponding to the high-resolution video frame samples comprises:

[0016] Collect high-resolution video samples and use threshold shot segmentation algorithm to obtain high-resolution video frame samples;

[0017] The high-resolution video frame samples are down-sampled using an image scaling algorithm to generate low-resolution video frame samples corresponding to the high-resolution video frame samples.

[0018] Optionally, the method further comprises:

[0019] Upsampling the to-be-processed video frame to obtain a sampled video frame;

[0020] The super-resolution video frame and the sampled video frame are added to obtain a target video frame.

[0021] Optionally, the method is applied to a terminal.

[0022] According to a second aspect of an embodiment of the present disclosure, a video super-resolution processing device is provided, comprising:

[0023] A video frame acquisition module is configured to acquire a video frame to be processed;

[0024] A processing module is configured to perform video super-resolution on the luminance component of the video frame to be processed and to perform interpolation processing on the chrominance component of the video frame to be processed through a video super-resolution model deployed on a neural network accelerator;

[0025] The integration module is configured to integrate the luminance component obtained after super-resolution and the chrominance component obtained after interpolation to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0026] According to a third aspect of an embodiment of the present disclosure, a video super-resolution processing device is provided, including:

[0027] processor;

[0028] a memory for storing processor-executable instructions;

[0029] Wherein, the processor is configured to:

[0030] Get the video frame to be processed;

[0031] Performing video super-resolution on the luminance component of the video frame to be processed and performing interpolation processing on the chrominance component of the video frame to be processed by using a video super-resolution model deployed on the neural network accelerator;

[0032] The luminance component obtained after super-resolution and the chrominance component obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0033] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the video super-resolution processing method provided in the first aspect of the present disclosure are implemented.

[0034] According to a fifth aspect of an embodiment of the present disclosure, a chip is provided, comprising a processor and an interface; the processor is used to read instructions to execute the video super-resolution processing method provided by the first aspect of the present disclosure.

[0035] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by deploying the video super-resolution model on the neural network accelerator, the video super-resolution of the brightness component of the video frame to be processed and the interpolation of the chrominance component of the video frame to be processed can be realized at the same time, so that the interpolation operation and the super-resolution of the Y channel data (i.e., the brightness component) can be solidified together, which not only does not affect the super-resolution of the Y channel data, but also enables the interpolation operation of the U and V channel data (i.e., the chrominance component) to be accelerated by calling the APU hardware resources. In addition, since the interpolation operation of the U and V channel data and the super-resolution of the Y channel data are simultaneously solidified in the video super-resolution model, the super-resolution of the Y channel data and the interpolation operation of the U and V channel data can be executed in parallel on the APU, thereby, on the basis of reducing the amount of calculation for the super-resolution of the Y channel data, the interpolation of the U and V channel data can be executed in parallel on the APU at the same time, so that a faster video super-resolution speed and lower power consumption can be obtained, the video is smooth and not stuck, and the operation power consumption is low, the phenomenon of heat generation is reduced, and the user experience is improved.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 The figure is a flowchart of a video super-resolution processing method according to an exemplary embodiment.

[0039] Figure 2 The figure is a schematic diagram of the structure of a video super-resolution model according to an exemplary embodiment.

[0040] Figure 3 The figure is a flowchart of a method for training a video super-resolution model according to an exemplary embodiment.

[0041] Figure 4 The figure is a block diagram of a video super-resolution processing device according to an exemplary embodiment.

[0042] Figure 5 The present invention is a block diagram of a video super-resolution processing device according to an exemplary embodiment.

[0043] Figure 6 The present invention is a block diagram of a video super-resolution processing device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0045] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0046] As discussed in the background technology, most of the current video super-resolution algorithms super-resolution the RGB three-channel data, or the YUV three-channel data. However, all three channels are calculated using deep learning algorithms, which requires a lot of calculations and is not friendly to terminals with limited computing power.

[0047] In order to reduce the amount of computation for video super-resolution, some algorithm neural networks now only perform video super-resolution on the Y channel data (i.e., brightness component) of the video frame, and perform traditional interpolation processing on the U and V channel data (i.e., chrominance component) of the video frame. Then, the output of the network is merged with the U and V channel data obtained after interpolation to obtain the final super-resolution video frame. The reason why this method is feasible is that the Y channel is the brightness channel and the UV is the color channel. Compared with color difference, human vision is more sensitive to brightness. Among them, YUV is a color encoding method. It is often used in various video processing components. When encoding photos or videos, YUV allows the bandwidth of chrominance to be reduced in consideration of human perception. "Y" represents brightness (Luminance or Luma), that is, grayscale value, and "U" and "V" represent chrominance (Chrominance or Chroma), which is used to describe the color and saturation of the image and is used to specify the color of the pixel.

[0048] By only performing video super-resolution on the Y channel data and interpolating the U and V channel data, the algorithm's computational load is greatly reduced. However, when the algorithm is deployed on the terminal, the Y channel data will call the APU hardware resources for super-resolution, while the interpolation operation of the U and V channel data can only be run on the CPU. The terminal's CPU calculation is very slow and has high power consumption. On the other hand, the deployment of the video player is based on the video post-processing (VPP) architecture. In this architecture, the interface data transmitted to the video super-resolution algorithm after the decoder is decoded is YUV data. Therefore, for data running on different computing resources, it can only be processed serially, that is, the Y channel data is first super-resolved on the APU, and then the U and V channel data are interpolated on the CPU, or a new thread is opened, that is, one thread performs video super-resolution on the Y channel data on the APU, and another thread interpolates the U and V channel data on the CPU. If serialization increases time consumption, and opening another thread will cause the terminal power consumption to increase, therefore, although this method reduces the computational load, it will still cause an increase in power consumption or running time when deployed on the terminal.

[0049] In view of this, the present disclosure provides a video super-resolution processing method, device, storage medium and chip.

[0050] Figure 1 FIG. 1 is a flowchart of a video super-resolution processing method according to an exemplary embodiment. The method can be applied to terminals such as smart phones and tablet computers, and can also be applied to servers. Figure 1 As shown, the above-mentioned video super-resolution processing method may include the following S101 to S103.

[0051] In S101, a video frame to be processed is obtained.

[0052] In the present disclosure, the video frame to be processed is a low-resolution video frame. In order to improve the clarity of the video frame, video super-resolution may be performed on it.

[0053] In S102, video super-resolution is performed on the luminance component of the video frame to be processed, and interpolation processing is performed on the chrominance component of the video frame to be processed through a video super-resolution model deployed on the neural network accelerator.

[0054] In the present disclosure, the neural network accelerator may be, for example, MediaTek's neural network accelerator neuropilot, Qualcomm's SNPE, etc. The luminance component of the video frame to be processed is the Y channel data of the video frame to be processed, and the chrominance component of the video frame to be processed includes the U channel data of the video frame to be processed and the V channel data of the video frame to be processed.

[0055] By deploying the video super-resolution model on the neural network accelerator, the video super-resolution of the brightness component (i.e., Y channel data) of the video frame to be processed and the interpolation of the chrominance component (i.e., U and V channel data) of the video frame to be processed can be realized simultaneously. In this way, the interpolation operation and the Y channel data super-resolution can be solidified together, and because the video super-resolution model is deployed on the neural network accelerator, the Y channel data super-resolution and the U and V channel data interpolation operations can call APU hardware resources for acceleration. Among them, the Y channel is the brightness channel, and UV is the color channel. Compared with color difference, human vision is more sensitive to brightness. Therefore, in order to reduce the amount of calculation, only the Y channel data is super-resolution, and the U and V channel data are interpolated.

[0056] In addition, since the interpolation operation and Y channel data super-resolution are simultaneously solidified in the video super-resolution model, the amount of calculation for Y channel data super-resolution is reduced, while the interpolation operation of U and V channel data originally executed on the CPU is run on the APU, that is, Y channel data super-resolution and U and V channel data interpolation operations can be executed in parallel on the APU.

[0057] In S103, the luminance component obtained after super-resolution and the chrominance component obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0058] In the present disclosure, the to-be-processed video frame and its corresponding super-resolution video frame are images of the same image with different resolutions.

[0059] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by deploying the video super-resolution model on the neural network accelerator, the video super-resolution of the brightness component of the video frame to be processed and the interpolation of the chrominance component of the video frame to be processed can be realized at the same time, so that the interpolation operation and the super-resolution of the Y channel data (i.e., the brightness component) can be solidified together, which not only does not affect the super-resolution of the Y channel data, but also enables the interpolation operation of the U and V channel data (i.e., the chrominance component) to be accelerated by calling the APU hardware resources. In addition, since the interpolation operation of the U and V channel data and the super-resolution of the Y channel data are simultaneously solidified in the video super-resolution model, the super-resolution of the Y channel data and the interpolation operation of the U and V channel data can be executed in parallel on the APU, thereby, on the basis of reducing the amount of calculation for the super-resolution of the Y channel data, the interpolation of the U and V channel data can be executed in parallel on the APU at the same time, so that a faster video super-resolution speed and lower power consumption can be obtained, the video is smooth and not stuck, and the operation power consumption is low, the phenomenon of heat generation is reduced, and the user experience is improved.

[0060] like Figure 2As shown, the video super-resolution model may include a video super-resolution network and an interpolation processing unit. The video super-resolution network is used to perform video super-resolution on the brightness component of the video frame to be processed; the interpolation processing unit is used to perform interpolation processing on the chrominance component of the video frame to be processed; then, the brightness component output by the video super-resolution network and obtained after super-resolution and the chrominance component output by the interpolation processing unit and obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0061] The video super-resolution network may be, for example, a deep neural network (DNN), a convolutional neural network (CNN), or the like.

[0062] The interpolation processing unit can use a traditional interpolation algorithm for video super-resolution to interpolate the chrominance components of the video frame to be processed. For example, interpolation algorithms such as nearest neighbor interpolation, bilinear interpolation, cubic spline interpolation, mean interpolation, median interpolation, etc. can be used to interpolate the chrominance components of the video frame to be processed. Among them, typical operations of the interpolation algorithm include: upsampling and downsampling of the image, and inserting new pixels between the original pixel points using a suitable interpolation algorithm based on the original image pixels.

[0063] In addition, the above video super-resolution model is Figure 3 The S301 and S302 shown in are trained.

[0064] In S301, high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples are obtained.

[0065] In the present disclosure, the high-resolution video frame sample and its corresponding low-resolution video frame sample are images of different definition of the same image.

[0066] In S302, model training is performed by taking the luminance component of the low-resolution video frame sample as the input of the video super-resolution network, taking the chrominance component of the low-resolution video frame sample as the input of the interpolation processing unit, and taking the luminance component of the high-resolution video frame sample as the target output of the video super-resolution network to obtain a video super-resolution model.

[0067] In the present disclosure, the interpolation processing unit may adopt the above-mentioned traditional interpolation algorithm. There is no need to adjust the parameters of the algorithm during the model training stage, but only the model parameters of the video super-resolution network need to be adjusted.

[0068] Specifically, the brightness component of the low-resolution video frame sample can be input into the video super-resolution network to obtain the super-resolution brightness component of the low-resolution video frame sample. At the same time, the chrominance component of the low-resolution video frame sample is input into the interpolation processing unit; then, according to the difference between the super-resolution brightness component of the low-resolution video frame sample and the brightness component of the high-resolution video frame sample, the model parameters of the video super-resolution network are updated; thereafter, new training data (i.e., high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples) are re-acquired to continue model training, i.e., return to the above S301 until the training cutoff condition is reached.

[0069] In one embodiment, the training cutoff condition may be that the number of trainings reaches a preset number of times, which may be set according to the actual usage scenario. When the number of trainings reaches the preset number of times, it may be determined that the number of trainings is sufficient so that the super-resolution processing capability of the super-resolution processing network meets the resolution requirements.

[0070] In another embodiment, the training cutoff condition may be that the target loss of the video super-resolution network is less than a preset threshold, and the preset threshold may be set according to the actual usage scenario. When the target loss of the video super-resolution network is less than the preset threshold, it can be considered that the super-resolution processing capability of the video super-resolution network meets the resolution requirement and can perform high-resolution super-resolution on the luminance component.

[0071] The following is a detailed description of the specific implementation of obtaining high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples in the above S301. Specifically, it can be implemented in a variety of ways. In one implementation, high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples can be obtained from an open source database containing multiple high- and low-resolution image pairs, wherein the high- and low-resolution image pairs include a high-resolution image and a low-resolution image corresponding to the high-resolution image.

[0072] In another embodiment, a high-resolution video frame sample may be obtained from an open source database containing multiple high-resolution images, and then the high-resolution video frame sample may be downsampled to obtain a low-resolution video frame sample corresponding to the high-resolution video frame sample.

[0073] In another embodiment, the high-resolution video frame samples and the low-resolution video frame samples corresponding to the high-resolution video frame samples may be obtained by the following steps (1) and (2):

[0074] Step (1): Collect high-resolution video samples and use the threshold shot segmentation algorithm to obtain high-resolution video frame samples.

[0075] In the present disclosure, a threshold shot segmentation algorithm may be used to perform shot segmentation on high-resolution video samples according to a preset key frame threshold to obtain high-resolution video frame samples.

[0076] Step (2): down-sample the high-resolution video frame samples using an image scaling algorithm to generate low-resolution video frame samples corresponding to the high-resolution video frame samples.

[0077] In the present disclosure, an image scaling algorithm such as the imresize function of MATLAB may be used to perform multiple times of bicubic downsampling on high-resolution video frame samples to obtain corresponding low-resolution video frame samples.

[0078] In addition, the above video super-resolution processing method may further include the following two steps:

[0079] Up-sampling the video frame to be processed to obtain a sampled video frame;

[0080] The super-resolution video frame and the sampled video frame are added to obtain the target video frame.

[0081] In the above implementation, the video super-resolution processing result (ie, the super-resolution video frame) is added to the sampled video frame, so that the video super-resolution effect of the video frame to be processed is better and the reliability of the video super-resolution processing result is improved.

[0082] Figure 4 FIG. 1 is a block diagram of a video super-resolution processing device according to an exemplary embodiment. Figure 4 , the device 200 comprises:

[0083] The video frame acquisition module 201 is configured to acquire a video frame to be processed;

[0084] The processing module 202 is configured to perform video super-resolution on the luminance component of the video frame to be processed and perform interpolation processing on the chrominance component of the video frame to be processed through a video super-resolution model deployed on the neural network accelerator;

[0085] The integration module 203 is configured to integrate the luminance component obtained after super-resolution and the chrominance component obtained after interpolation to obtain a super-resolution video frame corresponding to the video frame to be processed.

[0086] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by deploying the video super-resolution model on the neural network accelerator, the video super-resolution of the brightness component of the video frame to be processed and the interpolation of the chrominance component of the video frame to be processed can be realized at the same time, so that the interpolation operation and the super-resolution of the Y channel data (i.e., the brightness component) can be solidified together, which not only does not affect the super-resolution of the Y channel data, but also enables the interpolation operation of the U and V channel data (i.e., the chrominance component) to be accelerated by calling the APU hardware resources. In addition, since the interpolation operation of the U and V channel data and the super-resolution of the Y channel data are simultaneously solidified in the video super-resolution model, the super-resolution of the Y channel data and the interpolation operation of the U and V channel data can be executed in parallel on the APU, thereby, on the basis of reducing the amount of calculation for the super-resolution of the Y channel data, the interpolation of the U and V channel data can be executed in parallel on the APU at the same time, so that a faster video super-resolution speed and lower power consumption can be obtained, the video is smooth and not stuck, and the operation power consumption is low, the phenomenon of heat generation is reduced, and the user experience is improved.

[0087] Optionally, the video super-resolution model includes a video super-resolution network and an interpolation processing unit;

[0088] Wherein, the video super-resolution network is used to perform video super-resolution on the brightness component of the video frame to be processed;

[0089] The interpolation processing unit is used to perform interpolation processing on the chrominance component of the video frame to be processed.

[0090] Optionally, the video super-resolution model is obtained by training a model training device, wherein the model training device may include:

[0091] A sample acquisition module is configured to acquire high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples;

[0092] The training module is configured to perform model training by taking the luminance component of the low-resolution video frame sample as the input of the video super-resolution network, taking the chrominance component of the low-resolution video frame sample as the input of the interpolation processing unit, and taking the luminance component of the high-resolution video frame sample as the target output of the video super-resolution network to obtain the video super-resolution model.

[0093] Optionally, the sample acquisition module includes:

[0094] The acquisition submodule is configured to acquire high-resolution video samples and obtain high-resolution video frame samples by using a threshold shot segmentation algorithm;

[0095] The down-sampling submodule is configured to down-sample the high-resolution video frame samples by using an image scaling algorithm to generate low-resolution video frame samples corresponding to the high-resolution video frame samples.

[0096] Optionally, the device 200 further includes:

[0097] A sampling module is configured to upsample the video frame to be processed to obtain a sampled video frame;

[0098] The adding module is configured to add the super-resolution video frame and the sampling video frame to obtain a target video frame.

[0099] Optionally, the device 200 is applied to a terminal.

[0100] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the video super-resolution processing method provided by the present disclosure are implemented.

[0102] Figure 5 8 is a block diagram of a device 800 for video super-resolution processing according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0103] Reference Figure 5 , the device 800 may include one or more of the following components: a first processing component 802 , a first memory 804 , a first power supply component 806 , a multimedia component 808 , an audio component 810 , a first input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0104] The first processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The first processing component 802 may include one or more first processors 820 to execute instructions to complete all or part of the steps of the above-mentioned video super-resolution processing method. In addition, the first processing component 802 may include one or more modules to facilitate the interaction between the first processing component 802 and other components. For example, the first processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the first processing component 802.

[0105] The first memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The first memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0106] The first power source component 806 provides power to various components of the device 800. The first power source component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 800.

[0107] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0108] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the first memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0109] The first input / output interface 812 provides an interface between the first processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0110] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0111] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0112] In an exemplary embodiment, the device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned video super-resolution processing method.

[0113] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a first memory 804 including instructions, and the instructions can be executed by the first processor 820 of the device 800 to complete the above-mentioned video super-resolution processing method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0114] In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be an IC or a collection of multiple ICs; the chip can include but is not limited to the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), SOC (System on Chip, SoC), etc. The above-mentioned integrated circuit or chip can be used to execute executable instructions (or codes) to implement the above-mentioned video super-resolution processing method. The executable instructions can be stored in the integrated circuit or chip, or can be obtained from other devices or equipment, for example, the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instruction can be stored in the first memory, and when the executable instruction is executed by the processor, the above-mentioned video super-resolution processing method is implemented; alternatively, the integrated circuit or chip can receive the executable instruction through the interface and transmit it to the processor for execution, so as to implement the above-mentioned video super-resolution processing method.

[0115] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device. The computer program has a code portion for executing the above-mentioned video super-resolution processing method when executed by the programmable device.

[0116] Figure 6 1 is a block diagram of a device 1900 for video super-resolution processing according to an exemplary embodiment. For example, the device 1900 may be provided as a server. Figure 6 The apparatus 1900 includes a second processing component 1922, which further includes one or more second processors, and a memory resource represented by a second memory 1932 for storing instructions executable by the second processing component 1922, such as an application. The application stored in the second memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the second processing component 1922 is configured to execute instructions to perform the above-mentioned video super-resolution processing method.

[0117] The device 1900 may further include a second power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and a second input / output interface 1958. The device 1900 may operate based on an operating system stored in the second memory 1932, such as Windows Server 2000. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0118] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0119] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video super-resolution processing method, characterized in that: include: Get the video frame to be processed; Performing video super-resolution on the luminance component of the video frame to be processed and performing interpolation processing on the chrominance component of the video frame to be processed by using a video super-resolution model deployed on the neural network accelerator; The luminance component obtained after super-resolution and the chrominance component obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

2. The method according to claim 1, characterized in that The video super-resolution model includes a video super-resolution network and an interpolation processing unit; Wherein, the video super-resolution network is used to perform video super-resolution on the brightness component of the video frame to be processed; The interpolation processing unit is used to perform interpolation processing on the chrominance component of the video frame to be processed.

3. The method according to claim 2, characterized in that The video super-resolution model is trained in the following way: Acquire a high-resolution video frame sample and a low-resolution video frame sample corresponding to the high-resolution video frame sample; The video super-resolution model is obtained by performing model training by taking the luminance component of the low-resolution video frame sample as the input of the video super-resolution network, taking the chrominance component of the low-resolution video frame sample as the input of the interpolation processing unit, and taking the luminance component of the high-resolution video frame sample as the target output of the video super-resolution network.

4. The method according to claim 3, characterized in that The obtaining of high-resolution video frame samples and low-resolution video frame samples corresponding to the high-resolution video frame samples comprises: Collect high-resolution video samples and use threshold shot segmentation algorithm to obtain high-resolution video frame samples; The high-resolution video frame samples are down-sampled using an image scaling algorithm to generate low-resolution video frame samples corresponding to the high-resolution video frame samples.

5. The method according to claim 1, characterized in that The method further comprises: Upsampling the to-be-processed video frame to obtain a sampled video frame; The super-resolution video frame and the sampled video frame are added to obtain a target video frame.

6. The method according to any one of claims 1 to 5, characterized in that The method is applied to a terminal.

7. A video super-resolution processing device, characterized in that: include: A video frame acquisition module is configured to acquire a video frame to be processed; A processing module is configured to perform video super-resolution on the luminance component of the video frame to be processed and to perform interpolation processing on the chrominance component of the video frame to be processed through a video super-resolution model deployed on a neural network accelerator; The integration module is configured to integrate the luminance component obtained after super-resolution and the chrominance component obtained after interpolation to obtain a super-resolution video frame corresponding to the video frame to be processed.

8. A video super-resolution processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Get the video frame to be processed; Performing video super-resolution on the luminance component of the video frame to be processed and performing interpolation processing on the chrominance component of the video frame to be processed by using a video super-resolution model deployed on the neural network accelerator; The luminance component obtained after super-resolution and the chrominance component obtained after interpolation are integrated to obtain a super-resolution video frame corresponding to the video frame to be processed.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A chip, characterized in that: The method comprises a processor and an interface; the processor is used to read instructions to execute the method according to any one of claims 1 to 6.

Images