Video processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a video processing method, device, electronic device and storage medium. The method comprises: obtaining a quality score of an image in a video; determining a target image from the image according to the quality score; performing noise reduction processing on the target image and replacing the target image with the target image after noise reduction processing to obtain a preprocessed video; inputting the preprocessed video into a video super-resolution model for super-resolution processing, and outputting a processed video. The embodiment of the present disclosure not only takes into account the noise problem of low-resolution images in real scenes, but also selectively processes the noise according to the image quality in the video. After the obtained preprocessed video is input into the video super-resolution model, a higher-quality high-resolution video can be obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a video processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer vision technology, video super-resolution technology has emerged. Its goal is to restore video sequences from low resolution to high resolution, thereby filling in the missing details caused by the resolution change. In recent years, video super-resolution technology has been widely applied in fields such as mobile phone photography, medical imaging, and short videos.

[0003] In the related art, a video super-resolution model based on machine learning is proposed to restore high-resolution images from low-resolution videos. However, due to various reasons, the video quality restored by the video super-resolution technology in the related art is not high. Summary of the Invention

[0004] The present disclosure provides a video processing method, device, electronic device, and storage medium to at least address the problem of low video quality restored by video super-resolution technology in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, there is provided:

[0006] Get the quality score of the images in the video;

[0007] determining a target image from the images according to the quality scores;

[0008] Performing noise reduction processing on the target image, and replacing the target image with the target image after noise reduction processing to obtain a preprocessed video;

[0009] The pre-processed video is input into a video super-resolution model for super-resolution processing, and the processed video is output.

[0010] In a possible implementation, obtaining a quality score of an image in a video includes:

[0011] Get images from the video;

[0012] The image is input into an image quality assessment model, and the image quality assessment model outputs a noise category and a quality score of the image, wherein the image quality assessment model is trained using the correspondence between sample images and noise categories and quality scores.

[0013] In one possible implementation, the image quality assessment model is trained using the correspondence between sample images, noise categories, and quality scores, including:

[0014] Acquire a sample image set, where the sample image set includes a plurality of sample images labeled with noise categories and quality scores;

[0015] Inputting the sample image into an initial image quality assessment model to generate a prediction result;

[0016] Based on the difference between the prediction result and the annotated noise category and quality score, the training parameters of the initial image quality assessment model are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the image quality assessment model.

[0017] In a possible implementation, determining a target image from the images according to the quality scores includes:

[0018] Acquire images with quality scores lower than a preset threshold from the images;

[0019] The image with a quality score lower than a preset threshold is used as the target image.

[0020] In a possible implementation, determining a target image from the images according to the quality scores includes:

[0021] A target image is acquired from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image.

[0022] In a possible implementation, performing noise reduction on the target image and replacing the target image with the noise-reduced target image to obtain a preprocessed video includes:

[0023] The target image is smoothed and replaced with the smoothed target image to obtain a preprocessed video.

[0024] In a possible implementation, the video super-resolution model is obtained by:

[0025] Acquire a sample video set, where the sample video set includes a plurality of sample video pairs, where the sample video pairs include a first sample video and a second sample video that matches the content of the first sample video, and where a resolution of the first sample video is lower than a resolution of the second sample video;

[0026] Inputting the first sample video into an initial video super-resolution model to generate a prediction result;

[0027] Based on the difference between the prediction result and the second sample video, the training parameters of the initial video super-resolution model are iteratively adjusted until the difference meets a preset requirement, thereby obtaining the video super-resolution model.

[0028] According to a second aspect of an embodiment of the present disclosure, a video processing apparatus is provided, including:

[0029] An acquisition module is used to obtain the quality score of the image in the video;

[0030] a determination module, configured to determine a target image from the images according to the quality scores;

[0031] a noise reduction module, configured to perform noise reduction processing on the target image and replace the target image with the target image after noise reduction processing to obtain a preprocessed video;

[0032] The processing module is used to input the pre-processed video into a video super-resolution model for super-resolution processing, and output the processed video.

[0033] In a possible implementation, the acquisition module includes:

[0034] The first acquisition submodule is used to acquire images in the video;

[0035] The evaluation submodule is used to input the image into an image quality assessment model, and output the noise category and quality score of the image through the image quality assessment model, wherein the image quality assessment model is trained using the correspondence between sample images and noise categories and quality scores.

[0036] In a possible implementation, a first generating submodule is further included, and the first generating submodule includes:

[0037] an acquiring unit, configured to acquire a sample image set, wherein the sample image set includes a plurality of sample images annotated with noise categories and quality scores;

[0038] A prediction unit, configured to input the sample image into an initial image quality assessment model to generate a prediction result;

[0039] A generating unit is configured to iteratively adjust the training parameters of the initial image quality assessment model based on the difference between the prediction result and the annotated noise category and quality score until the difference meets preset requirements, thereby obtaining the image quality assessment model.

[0040] In a possible implementation, the determining module includes:

[0041] A second acquisition submodule is configured to acquire, from the images, images whose quality scores are lower than a preset threshold;

[0042] The first determination submodule is configured to take the image with a quality score lower than a preset threshold as a target image.

[0043] In a possible implementation, the determining module includes:

[0044] The second determining submodule is configured to obtain a target image from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image.

[0045] In a possible implementation, the noise reduction module includes:

[0046] The smoothing module is used to smooth the target image and replace the target image with the smoothed target image to obtain a preprocessed video.

[0047] In a possible implementation, a generation module is further included, and the generation module includes:

[0048] a third acquisition submodule, configured to acquire a sample video set, wherein the sample video set includes a plurality of sample video pairs, wherein the sample video pair includes a first sample video and a second sample video matching the content of the first sample video, and wherein the resolution of the first sample video is lower than the resolution of the second sample video;

[0049] A prediction submodule, configured to input the first sample video into an initial video super-resolution model to generate a prediction result;

[0050] The second generation submodule is used to iteratively adjust the training parameters of the initial video super-resolution model based on the difference between the prediction result and the second sample video until the difference meets the preset requirements, thereby obtaining the video super-resolution model.

[0051] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0052] processor;

[0053] a memory for storing instructions executable by the processor;

[0054] The processor is configured to execute the instructions to implement the video processing method as described in any one of the embodiments of the present disclosure.

[0055] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method as described in any one of the embodiments of the present disclosure.

[0056] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes instructions, and is characterized in that when the instructions are executed by a processor of an electronic device, the electronic device is able to execute the video processing method as described in any one of the embodiments of the present disclosure.

[0057] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In the embodiments of the present disclosure, before inputting the low-resolution video into the video super-resolution model, a target image with a lower quality score is selectively determined based on the quality score of the image in the video. Noise reduction processing is performed on the target image to eliminate the noise information in the target image frame; for the target image with a higher quality score, noise reduction processing is not performed, thereby retaining more useful image texture information. The embodiments of the present disclosure not only take into account the noise problem of low-resolution images in real scenes, but also selectively process the noise based on the image quality in the video. After the obtained pre-processed video is input into the video super-resolution model, a higher-quality high-resolution video can be obtained.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0060] Figure 1 The figure is a flowchart of a method for processing a video according to an exemplary embodiment.

[0061] Figure 2 The figure is a schematic structural diagram of an image quality assessment model according to an exemplary embodiment.

[0062] Figure 3 is a comparison diagram of quality scoring results according to an exemplary embodiment.

[0063] Figure 4 The figure is a structural diagram of a video super-resolution model according to an exemplary embodiment.

[0064] Figure 5 The figure is a flowchart of a method for processing a video according to an exemplary embodiment.

[0065] FIG6( a ) is a diagram showing the effect of a video processing method in the prior art.

[0066] FIG6( b ) is an effect diagram showing a method for processing a video according to an exemplary embodiment.

[0067] Figure 7 The figure is a block diagram of a video processing apparatus according to an exemplary embodiment.

[0068] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0069] Figure 9 The figure is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0071] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0072] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0073] Figure 1 FIG. 1 is a flow chart showing a method for processing a video according to an exemplary embodiment. Figure 1 As shown, the method is used in a terminal or a server and includes the following steps.

[0074] Step S101: Obtain quality scores of images in a video.

[0075] In the disclosed embodiment, the video is a video to be processed and may include a low-resolution video. The image may include each frame in the video or multiple images in a video frame. In the disclosed embodiment, methods for assessing image quality and obtaining a quality score may include various methods, such as subjective and objective evaluation methods. Subjective evaluation methods can determine image quality by normalizing observer scores, such as the dual-stimulus damage grading method, the dual-stimulus continuous quality grading method, and the single-stimulus continuous quality grading method. Objective evaluation methods may include full-reference image instruction evaluation, which compares the image to be evaluated with the original image to obtain error analysis, such as mean squared error (MSE) and peak signal-to-noise ratio (PSNR). Semi-reference image quality evaluation may also be included, which extracts only partial image information as a reference, such as methods based on original image features, wavelet-domain statistical models, and digital watermarking. Non-reference image quality assessment may also be included, such as algorithms tailored to distortion types or machine learning-based algorithms.

[0076] Step S103: determining a target image from the images according to the quality scores.

[0077] In the embodiment of the present disclosure, there may be multiple ways to determine the target image from the image. In one example, the image with the lowest quality score can be selected from the image as the target image. In another example, the image with a quality score lower than a preset value can be selected from the image as the target image. For example, the quality score is projected into a coordinate system, the horizontal axis is the image frame number corresponding to the quality score, and the vertical axis is the quality score. Different quality scores form a curve that fluctuates up and down. By adjusting the preset value, the minimum point of the trough in the curve or the point near the minimum value can be screened out, and the images corresponding to these minimum points or the points near them can be used as the target image. In another example, by comparison, the image with a quality score lower than the quality score of the previous frame image and the quality score of the next frame image can be screened out, and the image can be used as the target image.

[0078] Step S104 , performing noise reduction processing on the target image, and replacing the target image with the target image after noise reduction processing to obtain a preprocessed video.

[0079] In an embodiment of the present disclosure, a method for denoising a target image may include a spatial pixel feature denoising algorithm and a transform domain denoising algorithm. The spatial pixel feature denoising algorithm may include arithmetic mean filtering, Gaussian filtering, bilateral filtering, guided filtering, and NLM (Non-Local means) algorithm. The transform domain denoising algorithm may include: wavelet transform domain algorithm, BM3D denoising algorithm, etc. In one example, one or more of the above denoising algorithms may be used in combination to achieve a better denoising effect.

[0080] Step S105: input the pre-processed video into a video super-resolution model for super-resolution processing, and output the processed video.

[0081] In an embodiment of the present disclosure, the video super-resolution model is used to restore a high-resolution image from one or more low-resolution images. The generation method of the video super-resolution model may include alignment and non-alignment methods. In one example, the alignment method may employ motion compensation and motion estimation techniques. Motion estimation aims to extract inter-frame motion information, while motion compensation is used to perform inter-frame warping operations based on the inter-frame motion information to align them, thereby restoring a high-resolution image. In one example, the video super-resolution model may be trained using machine learning methods, such as a convolutional neural network-based method. Input frames are first aligned using a cascade and deformable alignment module, and then the aligned frames are fused using a spatiotemporal attention fusion module. The fused result is then input into a reconstruction module for feature extraction, followed by upsampling to obtain a residual image. This residual image is then added to the directly upsampled target frame to obtain the final high-resolution image. Non-alignment methods may include spatial non-alignment and spatiotemporal non-alignment. Spatial non-alignment uses the network to independently learn relevant information within a frame, thereby performing super-resolution reconstruction. Spatiotemporal non-alignment methods may include three-dimensional convolution, recurrent convolutional neural networks, and non-local methods.

[0082] In the disclosed embodiment, before feeding a low-resolution video into a video super-resolution model, a target image with a lower quality score is selectively determined based on the quality scores of the images in the video. Denoising the target image eliminates noise information within the target image frame. No denoising is performed on target images with higher quality scores, thereby retaining more useful image texture information. The disclosed embodiment not only considers the noise issues of low-resolution images in real scenes but also selectively processes the noise based on the image quality in the video. By feeding the resulting pre-processed video into the video super-resolution model, a higher-quality, high-resolution video can be obtained.

[0083] In a possible implementation, obtaining a quality score of an image in a video includes:

[0084] Get images from the video;

[0085] The image is input into an image quality assessment model, and the image quality assessment model outputs a noise category and a quality score of the image, wherein the image quality assessment model is trained using the correspondence between sample images and noise categories and quality scores.

[0086] In the embodiment of the present disclosure, the noise categories may include video codec noise, Gaussian noise, Poisson noise, salt and pepper noise, dark light noise, etc. By adding noise of more noise categories to the sample image, the sample image will be closer to the image in the real scene. In the embodiment of the present disclosure, the image quality assessment model is obtained by training through a deep learning method using the correspondence between sample images, noise categories, and quality scores. The sample images may include sample images with more noise categories added. The quality scores of the annotations of the sample images can be obtained in advance, such as manual scoring, for example, dividing the quality scores into 1-10 points.

[0087] Figure 2 FIG1 is a structural diagram of an image quality assessment model according to an exemplary embodiment. Figure 2 As shown, in the embodiment of the present disclosure, the image quality assessment model can be based on the VGG-16 network structure, which includes 13 convolutional layers, such as Figure 2 The dark rectangle indicates 5 pooling layers, as indicated by the white rectangle in the figure, and 3 fully connected layers. Figure 2 Not shown. According to the task of model training, the image quality score is predicted. Therefore, the probability of obtaining a quality score (for example, 1-10 points) of the sample image can be output separately in the last time of the network. Based on the predicted result and the true value, the difference between the two is calculated by the loss function, and the training parameters in the image quality assessment model are adjusted. It should be noted that the network structure of the image quality assessment model is not limited to the above examples. For example, the Inception-v2 network structure, the MobileNet network structure, and the ResNet-18 network structure can also be used as the network of the image quality assessment model. Inspired by the technical essence of this application, technical personnel in the relevant field may make other changes, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0088] In the embodiment of the present disclosure, the image quality assessment model can be obtained through a deep learning method. During the model training process, more sample images of noise categories are used to train the image quality assessment model, thereby improving the generalization ability of the model and obtaining a more accurate image quality score.

[0089] In one possible implementation, the image quality assessment model is trained using the correspondence between sample images, noise categories, and quality scores, including:

[0090] Acquire a sample image set, where the sample image set includes a plurality of sample images labeled with noise categories and quality scores;

[0091] Inputting the sample image into an initial image quality assessment model to generate a prediction result;

[0092] Based on the difference between the prediction result and the annotated noise category and quality score, the training parameters of the initial image quality assessment model are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the image quality assessment model.

[0093] In an embodiment of the present disclosure, the sample set includes multiple sample images labeled with noise categories and quality scores, for example, sample image a: Gaussian noise, score 3; sample image b: codec noise, score 6; and sample image c: motion estimation noise, score 7. In one example, the sample set can be divided into a training set, a validation set, and a test set according to a certain ratio. The sample images in the training set are input into an initial image quality assessment model, where the initial image quality assessment model can be any of the network structures described in the above embodiments. The sample images undergo feature extraction, pooling, and normalization in the initial image quality assessment model, ultimately predicting the probability of the sample image corresponding to the noise category and the probability of the corresponding quality score. Based on the difference between the predicted results and the labeled results in the training set, the training parameters in the initial image quality assessment model are iteratively adjusted until the difference meets preset requirements. In one example, the sample images in the validation set can be used to adjust the hyperparameters of the initial image quality model to obtain the final image quality assessment model. In another example, the sample images in the test set can be used to test the performance of the image quality assessment model and its accuracy.

[0094] Figure 3 is a comparison chart of quality score results according to an exemplary embodiment. Figure 3As shown, the horizontal axis represents the frame number of the image in the video, that is, the frame sequence number, which is used to indicate the frame number. The vertical axis represents the quality score. Data curve 301 is the result of performing noise processing on the sample image with a single noise and performing quality assessment on the processed sample image. Data curve 302 is the result of using the interpolation upsampling method to obtain a high-resolution sample image after the above processing, and performing quality assessment on the high-resolution sample image. Data curve 303 is the result of performing noise processing on the sample image with multiple noises and performing quality assessment on the processed sample image. Among them, the quality changes of data curve 301 and data curve 302 are relatively stable. The quality of different frames in data curve 303 varies greatly, and therefore it is closer to the real scene.

[0095] The disclosed embodiments provide a method for training an image quality assessment model. This method, through deep learning, not only predicts the noise category to which an image belongs but also accurately predicts the quality score for that noise category. By referencing different noise categories, the image quality assessment model has enhanced generalization capabilities and more accurate prediction results.

[0096] In a possible implementation, determining a target image from the images according to the quality scores includes:

[0097] Acquire images with quality scores lower than a preset threshold from the images;

[0098] The image with a quality score lower than a preset threshold is used as the target image.

[0099] In an embodiment of the present disclosure, based on the images in the video and the corresponding quality scores, images with lower quality scores can be screened out from the images by setting a preset threshold. In one example, by adjusting the size of the preset threshold, different numbers of target images can be obtained. For example: when the preset value is 0.4, group A target images are obtained, and when the preset value is 0.3, group B target images are obtained, wherein the number of group A target images may be different from the number of group B target images. In one example, the target images of group A and group B are subjected to denoising processing respectively, and are respectively input into a video super-resolution model to obtain processed videos, i.e., high-resolution videos corresponding to group A and high-resolution videos corresponding to group B, and the high-resolution videos corresponding to group A and group B are selected as the restored high-resolution videos.

[0100] The disclosed embodiment filters target images with low quality scores by setting a threshold, which has the beneficial effects of simple operation and easy implementation.

[0101] In a possible implementation, determining a target image from the images according to the quality scores includes:

[0102] A target image is acquired from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image.

[0103] In the embodiment of the present disclosure, the target image with the quality score at the minimum point can be screened out by comparison. In one example, for example, the frame numbers of the video are arranged as follows: 1 frame, 2 frames, 3 frames...20 frames. Among these frames, frame 3, frame 8, frame 12, frame 15, and frame 19 meet the conditions of the target image. Take frame 12 as an example, because the quality score of frame 12 is less than that of frame 11 and frame 13. In another example, reference Figure 3 As shown, there are 9 minimum points in the data curve 303. In one example, the target image meeting the above conditions is subjected to noise reduction processing and input into a video super-resolution model for super-resolution processing to obtain a processed video.

[0104] In this embodiment of the present disclosure, the quality score of the selected target image is lower than the quality score of the frame preceding and following the target image. This allows the present disclosure to filter out target images with lower quality scores while ensuring that the selected target images are not adjacent images. Because adjacent images contain a large amount of inter-frame information, performing noise processing simultaneously can easily remove useful image information. Therefore, this embodiment of the present disclosure can significantly improve the quality of high-resolution video restoration.

[0105] In a possible implementation, performing noise reduction on the target image and replacing the target image with the noise-reduced target image to obtain a preprocessed video includes:

[0106] The target image is smoothed and replaced with the smoothed target image to obtain a preprocessed video.

[0107] In the embodiment of the present disclosure, the smoothing process may include using a low-pass filtering algorithm to process the target image. In one example, the target image is smoothed using the following algorithm:

[0108]

[0109] Among them, C represents the smoothing module, represents the smoothed image, Express After smoothing the image again, Represents the image before smoothing, that is, Perform a smoothing process to obtain Send the image to the smoothing module to get the smoothed image. Calculate the absolute error of each pixel between the smoothed image and the input image. If the absolute error is greater than or equal to the preset value θ, the smoothed image needs to be smoothed again. For example, if but When the absolute error is less than the preset value θ, the smoothing process ends and the smoothing result is Replace the image x before processing i .

[0110] In the noise reduction process, the embodiment of the present disclosure selects a smoothing method, which has the beneficial effect of simple operation and easy implementation. By smoothing the target image, when the result after smoothing does not meet the preset requirements, the result after smoothing can be further smoothed, thereby ensuring the effectiveness of the smoothing result and ultimately improving the super-resolution effect.

[0111] Figure 4 FIG. 1 is a structural diagram of a video super-resolution model according to an exemplary embodiment. Figure 4 As shown, the method of obtaining the video super-resolution model includes:

[0112] Acquire a sample video set, where the sample video set includes a plurality of sample video pairs, where the sample video pairs include a first sample video and a second sample video that matches the content of the first sample video, and where a resolution of the first sample video is lower than a resolution of the second sample video;

[0113] Inputting the first sample video into an initial video super-resolution model to generate a prediction result;

[0114] Based on the difference between the prediction result and the second sample video, the training parameters of the initial video super-resolution model are iteratively adjusted until the difference meets a preset requirement, thereby obtaining the video super-resolution model.

[0115] In the embodiment of the present disclosure, the sample set includes multiple sample video pairs, such as sample video a1, sample video a2; sample video b2, sample video b2; sample video c1, sample video c2, etc., wherein sample video a1 and sample video a2 are videos with the same content, that is, the image content in the video corresponds one to one, and the resolution of sample video a1 is lower than the resolution of sample video a2; sample video b1 and sample video b2 are videos with the same content, and the resolution of sample video b1 is lower than the resolution of sample video b2; sample video c1 and sample video c2 are videos with the same content, and the resolution of sample video c1 is lower than the resolution of sample video c2.

[0116] In one example, the sample set can be divided into a training set, a validation set, and a test set according to a certain ratio. The sample images in the training set are input into the initial image quality assessment model, wherein the initial image quality assessment model may include the following: Figure 4 The bidirectional recurrent convolutional network shown. Figure 4 In the figure, step 401 indicates inputting a low-resolution video, step 402 indicates performing noise reduction on the low-resolution video, step 403 indicates inputting the noise-reduced video into a video super-resolution model based on a bidirectional recurrent convolutional network, and step 404 indicates obtaining a high-resolution video.

[0117] In an embodiment of the present disclosure, during model training, a first sample video undergoes feature extraction, pooling, normalization, and other processes in an initial video super-resolution model, and ultimately predicts a higher-resolution video. Based on the difference between the predicted result and the second sample video, the training parameters in the initial video super-resolution model are iteratively adjusted until the difference meets the preset requirements. In one example, the hyperparameters in the initial video super-resolution model can be adjusted using sample images in a validation set to obtain a final video super-resolution model. In one example, the performance of the video super-resolution model can be tested using sample images in a test set to test the accuracy of the model.

[0118] The disclosed embodiments provide a method for training a video super-resolution model based on a bidirectional recurrent convolutional network trained using deep learning methods. By adding a noise reduction step before high-resolution processing, the complexity of video super-resolution model training can be reduced, the model converges quickly, and prediction results are more accurate.

[0119] Figure 5 FIG. 1 is a flowchart of a method for processing a video according to an exemplary embodiment. Figure 5 As shown, the method includes:

[0120] Step S501: Acquire an image in a video.

[0121] In the embodiment of the present disclosure, the video is a video to be processed, and may include a low-resolution video. The image may include each frame of the video, or may include multiple images of the video frame.

[0122] Step S503: input the image into an image quality assessment model, and output the noise category and quality score of the image through the image quality assessment model.

[0123] In the disclosed embodiment, the noise categories may include video codec noise, Gaussian noise, Poisson noise, salt and pepper noise, and dark light noise. Adding noise from more noise categories to the sample images will make the sample images more similar to images in real scenes. In the disclosed embodiment, the image quality assessment model is trained using deep learning methods using the correspondence between sample images, noise categories, and quality scores. The sample images may include sample images with more noise categories added.

[0124] Step S505 : Acquire a target image from the image, wherein the quality score of the target image is determined to be lower than the quality score of the previous frame image and the quality score of the next frame image of the target image.

[0125] In the disclosed embodiments, a comparison can be used to select target images with a minimum quality score. In one example, the video frames are arranged as follows: 1, 2, 3, ... 20. Among these frames, frames 3, 8, 12, 15, and 19 meet the criteria for being a target image. Frame 12 is used as an example, as its quality score is lower than both frames 11 and 13.

[0126] Step S507 , smoothing the target image and replacing the target image with the smoothed target image to obtain a preprocessed video.

[0127] In the embodiment of the present disclosure, the smoothing process may include processing the target image using a low-pass filtering algorithm, and replacing the target image with the target image processed by the low-pass filtering to obtain a pre-processed video.

[0128] Step S107: input the pre-processed video into a video super-resolution model for super-resolution processing, and output the processed video.

[0129] In the embodiment of the present disclosure, the video super-resolution model is used to restore a high-resolution image from one or more low-resolution images. Figure 4 As shown, step 401 represents inputting a low-resolution video, step 402 represents performing noise reduction processing on the low-resolution video, step 403 represents inputting the noise-reduced video into a video super-resolution model based on a bidirectional recurrent convolutional network, and step 404 represents obtaining a high-resolution video.

[0130] FIG6(a) is a rendering of a video processing method in the prior art. FIG6(b) is a rendering of a video processing method according to an exemplary embodiment. Referring to FIG6(a), the image is a high-resolution video restored by upsampling the difference of the input low-resolution video. Referring to FIG6(b), the image is a rendering of the low-resolution video after video processing using the video processing method of the embodiment of the present disclosure. It can be seen that the white vertical lines of the method of the embodiment of the present disclosure are sharper than those of the prior art, while those in FIG6(a) are relatively blurry.

[0131] It should be understood that although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0132] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0133] Figure 7 The block diagram of a video processing device according to an exemplary embodiment is shown. The video processing device includes:

[0134] An acquisition module 701 is used to obtain a quality score of an image in a video;

[0135] A determination module 703 is configured to determine a target image from the images based on the quality scores;

[0136] A noise reduction module 705 is configured to perform noise reduction processing on the target image and replace the target image with the noise-reduced target image to obtain a preprocessed video;

[0137] The processing module 707 is used to input the pre-processed video into a video super-resolution model for super-resolution processing, and output the processed video.

[0138] In a possible implementation, the acquisition module includes:

[0139] The first acquisition submodule is used to acquire images in the video;

[0140] The evaluation submodule is used to input the image into an image quality assessment model, and output the noise category and quality score of the image through the image quality assessment model, wherein the image quality assessment model is trained using the correspondence between sample images and noise categories and quality scores.

[0141] In a possible implementation, a first generating submodule is further included, and the first generating submodule includes:

[0142] an acquiring unit, configured to acquire a sample image set, wherein the sample image set includes a plurality of sample images annotated with noise categories and quality scores;

[0143] A prediction unit, configured to input the sample image into an initial image quality assessment model to generate a prediction result;

[0144] A generating unit is configured to iteratively adjust the training parameters of the initial image quality assessment model based on the difference between the prediction result and the annotated noise category and quality score until the difference meets preset requirements, thereby obtaining the image quality assessment model.

[0145] In a possible implementation, the determining module includes:

[0146] A second acquisition submodule is configured to acquire, from the images, images whose quality scores are lower than a preset threshold;

[0147] The first determination submodule is configured to take the image with a quality score lower than a preset threshold as a target image.

[0148] In a possible implementation, the determining module includes:

[0149] The second determining submodule is configured to obtain a target image from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image.

[0150] In a possible implementation, the noise reduction module includes:

[0151] The smoothing module is used to smooth the target image and replace the target image with the smoothed target image to obtain a preprocessed video.

[0152] In a possible implementation, a generation module is further included, and the generation module includes:

[0153] a third acquisition submodule, configured to acquire a sample video set, wherein the sample video set includes a plurality of sample video pairs, wherein the sample video pair includes a first sample video and a second sample video matching the content of the first sample video, and wherein the resolution of the first sample video is lower than the resolution of the second sample video;

[0154] A prediction submodule, configured to input the first sample video into an initial video super-resolution model to generate a prediction result;

[0155] The second generation submodule is used to iteratively adjust the training parameters of the initial video super-resolution model based on the difference between the prediction result and the second sample video until the difference meets the preset requirements, thereby obtaining the video super-resolution model.

[0156] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0157] Figure 8 1 is a block diagram of an electronic device 800 for a method for processing a video according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0158] Reference Figure 8 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0159] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0160] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, optical disk, or graphene memory.

[0161] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0162] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0163] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0164] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0165] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or components of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0166] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0167] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0168] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to implement the above method.

[0170] Figure 9 FIG. 1 is a block diagram of an electronic device 900 for a method for processing a video according to an exemplary embodiment. For example, the electronic device 900 may be a server. Figure 9 The electronic device 900 includes a processing component 920, which further includes one or more processors, and a memory resource represented by a memory 922 for storing instructions executable by the processing component 920, such as an application. The application stored in the memory 922 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 920 is configured to execute the instructions to perform the above method.

[0171] The electronic device 900 may further include a power supply component 924 configured to perform power management of the electronic device 900, a wired or wireless network interface 926 configured to connect the electronic device 900 to a network, and an input / output (I / O) interface 928. The electronic device 900 may operate based on an operating system stored in the memory 922, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0172] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 922 including instructions, and the instructions can be executed by a processor of the electronic device 900 to perform the above method. The storage medium can be a computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0173] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by a processor of the electronic device 900 to implement the above method.

[0174] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0175] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: Get images from the video; Inputting the image into an image quality assessment model, and outputting the noise category and quality score of the image through the image quality assessment model, wherein the image quality assessment model is trained using the correspondence between sample images, noise categories, and quality scores; Determining a target image from the image based on the quality score; determining the target image from the image based on the quality score comprises: acquiring a target image from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image, respectively; Performing noise reduction processing on the target image, and replacing the target image with the target image after noise reduction processing to obtain a preprocessed video; The pre-processed video is input into a video super-resolution model for super-resolution processing, and the processed video is output.

2. The method according to claim 1, characterized in that The image quality assessment model is trained using the correspondence between sample images, noise categories, and quality scores, and includes: Acquire a sample image set, where the sample image set includes a plurality of sample images labeled with noise categories and quality scores; Inputting the sample image into an initial image quality assessment model to generate a prediction result; Based on the difference between the prediction result and the annotated noise category and quality score, the training parameters of the initial image quality assessment model are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the image quality assessment model.

3. The method according to claim 1, characterized in that The step of performing noise reduction on the target image and replacing the target image with the target image after noise reduction to obtain a preprocessed video includes: The target image is smoothed and replaced with the smoothed target image to obtain a preprocessed video.

4. The method according to any one of claims 1 to 3, characterized in that The video super-resolution model is obtained by: Acquire a sample video set, where the sample video set includes a plurality of sample video pairs, where the sample video pairs include a first sample video and a second sample video that matches the content of the first sample video, and where a resolution of the first sample video is lower than a resolution of the second sample video; Inputting the first sample video into an initial video super-resolution model to generate a prediction result; Based on the difference between the prediction result and the second sample video, the training parameters of the initial video super-resolution model are iteratively adjusted until the difference meets a preset requirement, thereby obtaining the video super-resolution model.

5. A video processing device, characterized in that: include: An acquisition module is used to obtain the quality score of the image in the video; The acquisition module includes: a first acquisition submodule for acquiring an image in a video; an evaluation submodule for inputting the image into an image quality assessment model, and outputting the noise category and quality score of the image through the image quality assessment model, wherein the image quality assessment model is trained using the correspondence between sample images, noise categories, and quality scores; a determination module, configured to determine a target image from the image based on the quality score; the determination module comprising: a second determination submodule, configured to obtain a target image from the image, wherein the quality score of the target image is determined to be lower than the quality score of an image frame preceding the target image and the quality score of an image frame following the target image, respectively; a noise reduction module, configured to perform noise reduction processing on the target image and replace the target image with the target image after noise reduction processing to obtain a preprocessed video; The processing module is used to input the pre-processed video into a video super-resolution model for super-resolution processing, and output the processed video.

6. The device according to claim 5, characterized in that Also included is a first generation submodule, the first generation submodule including: an acquiring unit, configured to acquire a sample image set, wherein the sample image set includes a plurality of sample images annotated with noise categories and quality scores; A prediction unit, configured to input the sample image into an initial image quality assessment model to generate a prediction result; A generating unit is configured to iteratively adjust the training parameters of the initial image quality assessment model based on the difference between the prediction result and the annotated noise category and quality score until the difference meets preset requirements, thereby obtaining the image quality assessment model.

7. The device according to claim 5, characterized in that The noise reduction module includes: The smoothing module is used to smooth the target image and replace the target image with the smoothed target image to obtain a preprocessed video.

8. The device according to any one of claims 5 to 7, characterized in that Also included is a generation module, the generation module including: a third acquisition submodule, configured to acquire a sample video set, wherein the sample video set includes a plurality of sample video pairs, wherein the sample video pair includes a first sample video and a second sample video matching the content of the first sample video, and wherein the resolution of the first sample video is lower than the resolution of the second sample video; A prediction submodule, configured to input the first sample video into an initial video super-resolution model to generate a prediction result; The second generation submodule is used to iteratively adjust the training parameters of the initial video super-resolution model based on the difference between the prediction result and the second sample video until the difference meets the preset requirements, thereby obtaining the video super-resolution model.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video processing method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video processing method according to any one of claims 1 to 4.

11. A computer program product comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to any one of claims 1 to 4.

Images