Video image quality enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides a video image quality enhancement method and device, electronic equipment and a storage medium, and the enhancement method comprises the steps: obtaining a source video, and constructing a corresponding source video image frame set according to the frame image of the source video; inputting the source video image frame set into a pre-trained multi-scale image quality enhancement model, and extracting an image quality enhancement image frame set; acquiring a corresponding image quality enhanced video according to the image quality enhanced image frame set; and carrying out resolution and code rate conversion on the image quality enhanced video to obtain a plurality of multi-scale enhanced videos with different resolutions and different code rates. Compared with the prior art, the multi-scale image quality enhancement model can obtain the video image residual errors of the source video under a plurality of different resolutions and different code rates, so that in the conversion process of the resolutions and the code rates, information in the video image residual errors can be combined only through one-time image quality enhancement operation, and the conversion efficiency of the source video is improved. And the multi-scale enhanced video can have higher quality.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, device, electronic device and storage medium for enhancing video quality. Background Art

[0002] Nowadays, live video broadcasting is becoming more and more common in modern society, especially in the fields of social media, e-sports and online education. In live video broadcasting, multiple resolutions and bit rates are usually set to meet the diverse needs of viewers and optimize the resource usage of service providers. Traditional methods for optimizing the image quality of live video broadcasting require image quality enhancement operations for each different bit rate, which consumes a lot of computing resources, resulting in reduced efficiency of image quality enhancement operations and affecting the audience's viewing experience. Summary of the invention

[0003] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a video quality enhancement method, device, electronic device and storage medium for solving the problem of being able to quickly enhance the quality of videos with multiple resolutions and bit rates while reducing the consumption of computing resources.

[0004] According to a first aspect of the present application, a method for enhancing video quality is provided, the method comprising:

[0005] Acquire a source video, and construct a corresponding source video image frame set according to the frame images of the source video;

[0006] Input the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract an image quality enhancement image frame set; the multi-scale image quality enhancement model is used to obtain the image quality enhancement image frame set of the source video at different resolutions and different bit rates through the source video image frame set; and obtain a corresponding image quality enhancement video according to the image quality enhancement image frame set;

[0007] The image quality enhanced video is subjected to resolution and bit rate conversion to obtain a plurality of multi-scale enhanced videos with different resolutions and different bit rates.

[0008] Optionally, the training of the multi-scale image quality enhancement model specifically includes:

[0009] Obtain a number of training video data with different resolutions and the same content;

[0010] According to a number of time points, corresponding image frames are extracted from each of the training video data, and all the image frames extracted from each of the training video data are used to construct a training image frame set corresponding to the training video data; the training image frame sets are respectively input into a pre-trained single-scale image quality enhancement model to obtain a corresponding training enhanced image frame set;

[0011] Converting all the acquired training enhanced image sets to the same target resolution and superimposing them to obtain a multi-scale training image frame set;

[0012] Inputting the training image frame set corresponding to the target resolution into the multi-scale image quality enhancement model to obtain a multi-scale predicted image frame set;

[0013] The multiscale loss of the multiscale training image frame set and the multiscale prediction image frame set is calculated, and the multiscale image quality enhancement model is updated according to the multiscale loss.

[0014] Optionally, the step of inputting the plurality of training image frame sets into a pre-trained single-scale image quality enhancement model to obtain corresponding training enhanced image frame sets specifically includes:

[0015] Extracting enhanced residual image sets corresponding to a number of training image frame sets respectively through the single-scale image quality enhancement model;

[0016] The images in the enhanced residual image set are superimposed with the corresponding images in the training image frame set to obtain the training enhanced image frame set.

[0017] Optionally, the image in the enhanced residual image set is superimposed with the corresponding image in the training image frame set, which is expressed as follows:

[0018] HR r =LR r +M(LR r )*ResidualFactor r

[0019] In the formula, LR r represents the image in the training image frame set with resolution r, HR r represents the image in the training enhanced image frame set with a resolution of r obtained after superposition, M(LR r ) represents the LR extracted by the single-scale image quality enhancement model r The corresponding enhanced residual image, ResidualFactor r Represents the residual enhancement coefficient with resolution r.

[0020] Optionally, the training of the single-scale image quality enhancement model specifically includes:

[0021] Obtain a number of high-definition training video data with different resolutions and the same content;

[0022] Keeping the resolution unchanged, respectively reducing the bit rates of the high-definition training video data with different resolutions to obtain corresponding low-definition training video data;

[0023] According to a number of time points, corresponding image frames are extracted from the high-definition training video data and the low-definition training video data, respectively, and a high-definition training image frame set and a low-definition training image frame set are constructed according to all the image frames extracted from the high-definition training video data and the low-definition training video data, respectively;

[0024] Inputting the low-definition training image frame set into the single-scale image quality enhancement model to obtain a corresponding single-scale prediction image frame set;

[0025] The enhancement loss of the single-scale predicted image frame set and the corresponding high-definition training image frame set is calculated, and the single-scale image quality enhancement model is updated according to the enhancement loss.

[0026] Optionally, the single-scale image quality enhancement models include several ones; each of the single-scale image quality enhancement models is trained for one resolution.

[0027] Optionally, the single-scale image quality enhancement model includes a plurality of residual blocks;

[0028] The processing flow of the single-scale image quality enhancement model specifically includes:

[0029] Performing convolution processing on the input image to obtain convolution features of the input image;

[0030] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0031] Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks;

[0032] The residual extraction feature output by the last residual block is used as the residual image of the input image.

[0033] According to a second aspect of the present application, a video quality enhancement device is provided, the enhancement device comprising:

[0034] A video acquisition processing module, used to acquire a source video and construct a corresponding source video image frame set according to the frame images of the source video;

[0035] The image quality enhancement module is used to input the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract the image quality enhanced image frame set; the multi-scale image quality enhancement model is used to obtain the image quality enhanced image frame set according to the video image residuals of the source video at several different resolutions corresponding to the source video image frame set;

[0036] A video restoration module, used for obtaining a corresponding quality-enhanced video according to the quality-enhanced image frame set;

[0037] The video conversion module is used to convert the resolution and bit rate of the image quality enhanced video to obtain a plurality of multi-scale enhanced videos with different resolutions and different bit rates.

[0038] According to a third aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video quality enhancement method described in the first aspect above.

[0039] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer-readable program is stored. When the computer-readable program is executed, a video quality enhancement method described in the first aspect is implemented.

[0040] According to any one of the aspects provided above, a video quality enhancement method, device, and electronic device storage medium provided by the present application perform quality enhancement processing on a source video image frame set through a pre-trained multi-scale image quality enhancement model, wherein the multi-scale image quality enhancement model obtains a corresponding image quality enhancement image frame set based on the video image residuals of the source video at several different resolutions and different bit rates, and can fully perform feature learning based on the video image residuals of different resolutions and bit rates; at the same time, in the process of image quality enhancement, only one image quality enhancement operation is performed on the source video frame image set of the source video, so that multiple videos of different gears can be obtained quickly and effectively;

[0041] At the same time, in a video quality enhancement method, device, electronic device and storage medium provided by this embodiment, the single-scale image quality enhancement model extracts a residual image enhanced by the input image, and superimposes the residual image with the input image, so that the single-scale image quality enhancement model can effectively learn the enhanced residual information in the residual image; and by training the single-scale image quality enhancement model with videos of different resolutions and different bit rates, the single-scale image quality enhancement model can effectively enhance the image quality of videos with different resolutions and bit rates;

[0042] Furthermore, in a video quality enhancement method, device, electronic device and storage medium provided by the present embodiment, the enhanced residual images of each resolution obtained in the single-scale image quality enhancement model are learned through a multi-scale image quality enhancement model, so that enhancement information at different resolutions can be obtained, and thus the obtained image quality enhanced video can effectively include enhancement conditions at various resolutions and different bit rates. Finally, when the image quality enhanced video is converted in resolution and bit rate, the enhancement information can be retained, so that the obtained multi-scale enhanced video has a higher image quality effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A schematic diagram of an application scenario of the enhancement method provided in this embodiment.

[0045] Figure 2 This is a flowchart of the steps of the enhancement method provided in this embodiment.

[0046] Figure 3 A flowchart of the steps for training the multi-scale image quality enhancement model provided in this embodiment.

[0047] Figure 4 A flowchart of the steps for training a single-scale image quality enhancement model provided in this embodiment.

[0048] Figure 5 This is a structural diagram of the enhancement device provided in this embodiment.

[0049] Figure 6 This is a structural diagram of an electronic device provided in this embodiment.

[0050] The accompanying drawings are marked with: server 100, terminal 200, video acquisition processing module 11, image quality enhancement module 12, video restoration module 13, video conversion module 14, multi-scale model training module 15, single-scale model training module 16, memory 21, processor 22, bus 23, communication interface 24. DETAILED DESCRIPTION

[0051] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0052] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] Nowadays, live video streaming is becoming more and more common in modern society, especially in areas such as social media, e-sports, and online education.

[0055] Among them, the network conditions of different viewers vary greatly. Some viewers may use high-speed networks such as high-speed broadband, high-speed Wi-Fi or 5G networks, which enables these viewers to watch higher-resolution, higher-quality and smoother live videos, while others may rely on low-speed networks such as 4G networks or public Wi-Fi. These viewers may pay more attention to the smoothness of live video. In addition to the influence of network factors, the viewing devices used by viewers will also affect the viewing experience of viewers. The viewing devices used by viewers range from large HD TV screens to small smartphone screens. Different display screens have different display effects and will also bring different viewing experiences to viewers.

[0056] Therefore, in live video broadcasting, a variety of resolutions and bit rates are provided to viewers, so that the diverse needs of viewers can be met and the resource usage of service providers can be optimized. Usually, the highest resolution in live video broadcasting is the real-time transmitted video stream, which has not undergone any encoding compression, while other resolutions and bit rates are obtained through transcoding. For example, the resolutions and bit rates of mainstream live video broadcasting may include 1080P 10M, 1080P 4M, 720P 2M, and 540P 500K, of which the highest resolution 1080P 10M is the real-time transmitted video stream, and other resolutions and bit rates are obtained through transcoding of the 1080P10M video stream.

[0057] In the prior art, after transcoding, in order to provide viewers with a better viewing experience at each level, image quality enhancement operations are performed on the video at each level after transcoding. However, as the level of live video increases, these image quality enhancement operations consume a large amount of computing resources, making live video processing complex and inefficient.

[0058] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0059] For example, a schematic diagram of an application scenario of a video quality enhancement method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal 200 has a streaming media playback function and can also have an image processing function.

[0060] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0061] In one practicable manner, the server 100 and the terminal 200 may respectively execute a video quality enhancement method provided in an embodiment of the present application, or, optionally, the video quality enhancement method provided in an embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0062] like Figure 2 As shown, this embodiment provides a method for enhancing video quality, and the enhancement method includes:

[0063] S1: Acquire a source video, and construct a corresponding source video image frame set according to frame images of the source video;

[0064] In this embodiment, the source video can be obtained according to the actual live video stream, and the resolution and bit rate of the source video can be set according to the highest or most commonly used resolution and bit rate of the existing live video source. Because the actual image quality enhancement operation is performed by a pre-trained neural network model, and the pre-trained neural network model in this embodiment can perform image quality enhancement for different resolutions and bit rates, the highest or most commonly used resolution and bit rate of the existing live video source are used for setting the source video collection. As mentioned above, because the highest or most commonly used resolution and bit rate are real-time transmitted video streams, there is no need for transcoding operations, and collection can be more convenient and direct.

[0065] For example, if the highest or most commonly used resolution and bit rate is 1080P 10M, the resolution and bit rate of the source video may correspond to 1080P 10M.

[0066] S2: inputting the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract an image quality enhanced image frame set;

[0067] In this embodiment, the multi-scale image quality enhancement model is used to obtain the image quality enhanced image frame set of the source video at different resolutions and different bit rates through the source video image frame set.

[0068] Specifically, during the image quality enhancement operation of the source video, the multi-scale image quality enhancement model is pre-trained, that is, the training of the multi-scale image quality enhancement model can be performed offline to reduce the efficiency of the enhancement method.

[0069] In this embodiment, the multi-scale image quality enhancement model can adopt a common and mature image super-resolution network structure in the field of image quality enhancement and super-resolution, such as the ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) network; the ESRGAN network has a good image quality enhancement effect, but the network structure of the existing ESRGAN network is very complex, resulting in a large amount of calculation and parameters. In order to meet the requirements of live video broadcasting, in this embodiment, the basic structure of the existing ESRGAN network is trimmed. The specific trimming strategy is as follows:

[0070] Reducing the number of residual blocks: Specifically, the number of residual blocks in the existing ESRGAN network is reduced from 16 to 10. The residual blocks are used in the network to capture the details and features of the image, but too many residual blocks will increase the amount of calculation of the model, making it difficult to apply the model in real-time scenarios. Therefore, in this embodiment, the amount of calculation of the ESRGAN network is reduced by reducing the number of residual blocks, thereby improving the training and calculation efficiency of the multi-scale image quality enhancement model.

[0071] Reducing the number of channels: Specifically, the number of channels of the existing ESRGAN network is reduced from 256 to 64, that is, reduced to 1 / 4 of the original number. The number of channels in the ESRGAN network represents the number of filters used in each convolution operation in the network. In an embodiment, reducing the number of channels in the ESRGAN network helps to significantly reduce the demand for computing resources while retaining the necessary image detail information as much as possible.

[0072] Through the pruned ESRGAN network, the training of the multi-scale image quality enhancement model can be achieved more efficiently;

[0073] Specifically, Figure 3 As shown, the training of the multi-scale image quality enhancement model may include:

[0074] A1: Obtain several training video data with different resolutions and the same content;

[0075] As we all know, there are differences in image quality details between videos of different resolutions. For example, high-quality image quality details at 1080P resolution may not be fully retained after conversion to 720P and 540P resolutions, and vice versa. These differences are not only reflected in video acquisition and encoding, but also affect the way of image quality enhancement. In fact, as mentioned above, the image quality enhancement method applicable to 1080P resolution may lose a large degree of enhanced image quality details when the resolution is reduced to 720P or 540P. Similarly, when the image quality enhancement methods of 720P and 540P are applied to 1080P resolution, they may not be fine enough due to too many details, resulting in unclear enhancement effects or noise.

[0076] In order to avoid excessive consumption of computing resources and cover multiple different resolutions and bit rate levels at the same time, the multi-scale image quality enhancement model in this embodiment can effectively achieve significant image quality improvement at different resolutions. In order to effectively train the multi-scale image quality enhancement model so that the multi-scale image quality enhancement model can better adapt to various resolutions, it is necessary to collect a number of training video data with different resolutions and the same content. Specifically, live video streams with different resolutions can be collected as the training video data, wherein the collected training video data is preferably low-definition video data with lower picture quality, so as to better realize the enhanced learning of the multi-scale image quality enhancement model; in addition, the collection of the training video data can also be based on collecting low-definition video data of one resolution, and the basic low-definition video data is converted into a resolution to obtain low-definition video data of other resolutions.

[0077] A2: extracting corresponding image frames from each of the training video data according to a number of time points, and constructing a training image frame set corresponding to the training video data from all the image frames extracted from each of the training video data;

[0078] Since the contents of several of the training video data are the same, the corresponding frames at the same time point are the same. By extracting frames at the same time point in each of the training video data, corresponding image frames with different resolutions in several of the training video data can be obtained, and then all the image frames extracted from each of the training video data are used to construct a corresponding training image frame set. This can ensure that in the subsequent learning process, the multi-scale image quality enhancement model can learn the image details of images with the same content at different resolutions, and then can fully express the image details of different resolutions in the predicted image in the subsequent prediction process.

[0079] A3: inputting the plurality of training image frame sets into the pre-trained single-scale image quality enhancement model respectively to obtain corresponding training enhanced image frame sets;

[0080] As mentioned above, simply enhancing the image quality cannot make the enhanced image quality inherit the detail performance in other different resolutions. Therefore, in this embodiment, the multi-scale image quality enhancement model is not based on the learning prediction of each enhanced image, but through learning the enhancement information of each resolution, specifically, learning through the enhanced residual image of each resolution image. The enhanced residual image of each resolution can well reflect the enhancement information of the image at the corresponding resolution, so that the image obtained by the multi-scale image quality enhancement model can well preserve the details after each resolution enhancement.

[0081] In this embodiment, the enhanced residual images of each resolution are learned and extracted through the single-scale image quality enhancement model. The single-scale image quality enhancement module can be constructed based on the existing super-resolution model structure, but, as mentioned above, in this embodiment, the single-scale image quality enhancement model does not need to output the enhanced image, but needs to output the enhanced residual image, so this embodiment improves the structure of the existing super-resolution model, and the specific improvements may include:

[0082] The output layer network of the existing super-resolution model is removed, the residual block structure in the super-resolution model is retained, and the enhanced residual image is obtained through the residual block output. The processing flow of the single-scale image quality enhancement model may specifically include:

[0083] Performing convolution processing on the input image to obtain convolution features of the input image;

[0084] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0085] Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks;

[0086] The residual extraction feature output by the last residual block is used as the residual image of the input image.

[0087] Correspondingly, in this embodiment, the training process of the single-scale image quality enhancement model may include:

[0088] B1: Obtain several high-definition training video data with different resolutions and the same content;

[0089] In this embodiment, live video streams of different resolutions can be collected as the high-definition training video data, wherein the collected high-definition training video data is preferably high-definition video data with higher picture quality and bit rate. Different from the multi-scale image quality enhancement model, in the multi-scale image quality enhancement model, the corresponding enhancement information is obtained after the low-definition image is enhanced by the single-scale image quality enhancement model, and the training of the single-scale image quality enhancement model requires learning the process of image quality enhancement, and the high-definition video data is required as a basis for comparison, so as to better realize the enhanced learning of the single-scale image quality enhancement model; in addition, the collection of the high-definition training video data can also be obtained by collecting high-definition video data of one resolution as a basis, and converting the resolution of the basic high-definition video data to obtain high-definition video data of other resolutions.

[0090] B2: Keeping the resolution unchanged, respectively reducing the bit rates of the high-definition training video data with different resolutions to obtain corresponding low-definition training video data;

[0091] Specifically, in this embodiment, the high-definition training video data can be re-encoded by coding degradation to generate low-definition video data. After collecting high-quality, high-bitrate video data, a coding degradation operation needs to be performed to generate low-definition video data at the corresponding resolution. Coding degradation refers to re-encoding the source video according to the target bit rate, so that the video quality at different bit rates commonly used in practical applications can be simulated. Exemplarily, for 1080P resolution high-definition video data, coding is used to degenerate into low-definition video data of 1080P 10M and 1080P 4M, for 720P resolution high-definition video data, it is degraded into 720P 2M low-definition video data, and for 520P resolution high-definition video data, it is degraded into 540P 500K low-definition video data.

[0092] B3: extracting corresponding image frames from the high-definition training video data and the low-definition training video data according to a number of time points, and constructing a high-definition training image frame set and a low-definition training image frame set according to all the image frames extracted from the high-definition training video data and the low-definition training video data;

[0093] Since the corresponding high-definition training video data and low-definition training video data differ only in bit rate and video quality, and the video content is the same, the corresponding image frames can be extracted at the same time point to obtain the corresponding high-definition training image frame set and low-definition training image frame set. It can be understood that the main difference between the high-definition training image frame set and the low-definition training image frame set lies in the quality of the image, and the corresponding image content is the same, so as to facilitate loss training.

[0094] B4: inputting the low-definition training image frame set into the single-scale image quality enhancement model to obtain a corresponding single-scale prediction image frame set;

[0095] As described above, it can be understood that in this embodiment, after each image frame in the low-definition training image frame set is input into the single-scale image quality enhancement model, the single-scale image quality enhancement model enhances the image frame and outputs an enhanced residual image corresponding to each image frame, and superimposes the output enhanced residual image with each input image frame to obtain an output single-scale predicted image frame, and then obtains the single-scale predicted image frame set corresponding to each of the low-definition training image frame sets, and the single-scale predicted image frame contains enhanced residual information learned by the single-scale image quality enhancement model.

[0096] B5: Calculate the enhancement loss of the single-scale predicted image frame set and the corresponding high-definition training image frame set, and update the single-scale image quality enhancement model according to the enhancement loss.

[0097] As described above, each image frame in the single-scale predicted image frame set obtained in step B4 contains corresponding enhanced residual information, and these enhanced residual information need to be specifically learned. However, before the single-scale image quality enhancement model has completed training, its output result has not converged, that is, the image quality enhancement has not been achieved. Therefore, it is necessary to provide an enhanced "target" for the output of the single-scale image quality enhancement model so that the model converges to this "target" during the training process;

[0098] In this embodiment, since the low-definition training image frame set input into the single-scale image quality enhancement model is obtained by encoding and degrading the corresponding high-definition image quality enhancement model, the high-definition training image frame set corresponding to the low-definition training image frame set can well serve as the "target" of the single-scale image quality enhancement model. By calculating the enhancement loss of the high-definition prediction image frame set and the corresponding high-definition training image frame set, the single-scale image quality enhancement model can better learn to enhance the residual information, and finally obtain the single-scale image quality enhancement model that can output enhanced detail information with different resolutions.

[0099] It can be understood that in a preferred implementation of the present embodiment, the single-scale image quality enhancement model can learn and predict video data of multiple different resolutions. For example, video data of multiple different resolutions can be input into a single-scale image quality enhancement model for learning, and the obtained single-scale image quality enhancement model can predict video data of multiple different resolutions. In this implementation, the single-scale image quality enhancement model has higher versatility and does not need to distinguish between video data of different resolutions. It can be directly predicted by the single-scale image quality enhancement model.

[0100] In another preferred implementation of the present embodiment, multiple single-scale image quality enhancement models can be set, and the setting data of the single-scale image quality enhancement model matches the type of resolution and bit rate of the video data to be predicted, that is, one single-scale image quality enhancement model only predicts video data with one resolution and one bit rate. In this implementation, the single-scale image quality enhancement model is targeted and can more accurately predict the video data it targets, thereby effectively improving the accuracy of the prediction.

[0101] At the same time, the above two implementation modes can be combined, that is, the same single-scale image quality enhancement model can be used to predict video data within a certain resolution range and bit rate range, and for video data with a certain specific resolution and bit rate, a single-scale image quality enhancement model can be used to predict it separately. The specific settings can be made according to the actual application scenario.

[0102] Furthermore, the loss function of the enhancement loss may adopt mean square error, peak signal-to-noise ratio, structural similarity index, etc., which may be specifically set according to actual needs and is not further limited here.

[0103] Therefore, based on the processing and training process of the single-scale image quality enhancement model, it can be understood that in this embodiment, step A3 may specifically include:

[0104] Extracting enhanced residual image sets corresponding to a number of training image frame sets respectively through the single-scale image quality enhancement model;

[0105] The images in the enhanced residual image set are superimposed with the corresponding images in the training image frame set to obtain the training enhanced image frame set.

[0106] The step of superimposing the image in the enhanced residual image set with the corresponding image in the training image frame set may be specifically performed as follows:

[0107] HR r =LR r +M(LR r )*ResidualFactor r

[0108] In the formula, LR r represents the image in the training image frame set with resolution r, HR r represents the image in the training enhanced image frame set with a resolution of r obtained after superposition, M(LR r ) represents the LR extracted by the single-scale image quality enhancement model rThe corresponding enhanced residual image, ResidualFactor r represents the residual enhancement coefficient with a resolution of r, and the setting range of the residual enhancement coefficient is 0 to 1.

[0109] In this embodiment, the specific setting of the residual enhancement coefficient can be achieved through the output of the residual block in the corresponding single-scale image quality enhancement model, which is used to adjust the superposition of different resolution scales, and then adjust the intensity of image quality enhancement. When the residual enhancement coefficient is set to 1.0, it means that the default full enhancement effect is applied. When the residual enhancement coefficient is set to 0.2, it means that the enhancement effect is relatively weak, which is suitable for scenes requiring slight image quality enhancement.

[0110] A4: converting all the acquired training enhanced image sets to the same target resolution and superimposing them to obtain a multi-scale training image frame set;

[0111] Specifically, the target resolution to which the training enhanced image is converted can be selected from the resolution of the training video data, mainly to obtain video data that is convenient for loss training of the multi-scale image quality enhancement model. Because the training enhanced image set contains enhanced residual information, the enhanced residual information can be retained even if the resolution conversion is performed.

[0112] Preferably, the selected target resolution can be the same as the resolution of the source video, so that the multi-scale image quality enhancement model can directly perform prediction processing on the source video.

[0113] For example, if the training enhanced image set includes three resolutions of 1080P, 720P and 540P, and the target resolution can be selected as 1080P, then the overlay operation can be specifically performed as follows:

[0114] HR MULTI =LR 1080P +M 1080P (LR 1080P )*ResidualFactor 1080P

[0115] +Resize 1080P {M 720P (LR 720P )*ResidualFactor 720P}

[0116] +Resize 1080P {M 540P (LR 540P )*ResidualFactor 540P}

[0117] In the formula, HR MULTI Represents the superimposed multi-scale training image frame, Resize 1080P Indicates converting the image resolution to 1080P. 1080P (LR 1080P ), M 720P (LR 720P ) and M 540P (LR 540P ) represent the extraction of LR by the single-scale image quality enhancement model r The enhanced residual image of the corresponding resolution, such as M 1080P (LR 1080P ) represents extracting an enhanced residual image of an image in the training enhanced image set with a resolution of 1080P, and the resolution of the enhanced residual image is 1080P:

[0118] A5: inputting the training image frame set corresponding to the target resolution into the multi-scale image quality enhancement model to obtain a multi-scale predicted image frame set;

[0119] It can be understood that the training image frame set corresponding to the target resolution includes low-definition images of the target resolution. The low-definition images of the target resolution are input into the multi-scale image quality enhancement model to be trained, and the predicted images of the target resolution can be output. At this time, since the multi-scale image quality enhancement model has not been trained, the output result cannot meet the requirements of image quality enhancement.

[0120] A6: Calculate the multi-scale loss of the predicted image frame set and the training enhanced image frame set, and update the multi-scale image quality enhancement model according to the multi-scale loss.

[0121] The multi-scale predicted image frame set obtained in step A5 does not obtain the enhancement information mentioned in each single-scale image quality enhancement model because the multi-scale image quality enhancement model has not completed training. Similar to the single-scale image quality enhancement model, it is necessary to provide the multi-scale image quality enhancement model with a "target" that can contain enhancement information of all resolutions and bit rates, so that the multi-scale image quality enhancement model converges to this "target" during the training process;

[0122] In this embodiment, the multi-scale training image frame set is obtained by superimposing all the training enhanced image sets predicted by the single-scale image quality enhancement model, which includes enhancement information of all different resolutions and bit rates. Taking the multi-scale training image frame set as the "target" of the multi-scale image quality enhancement model can enable the multi-scale image quality enhancement model to fully learn the enhancement information of various resolutions and bit rates during the learning process, and finally enable the multi-scale predicted image set output by the multi-scale image quality enhancement model to include image quality enhancement information at various resolutions and bit rates. Specifically, in this embodiment, the loss function of the multi-scale loss can adopt mean square error, peak signal-to-noise ratio, and structural similarity index, etc., which can be set according to actual needs and are not further limited here.

[0123] It can be understood that the resolution range applicable to the multi-scale image quality enhancement model is related to the resolution range applicable to the single-scale image quality enhancement model. Therefore, during the training process of the single-scale image quality enhancement model, video data of multiple different resolutions can be used to train the single-scale image quality enhancement model, thereby improving the applicable scope of the single-scale image quality enhancement model. Similarly, the applicable scope of the multi-scale image quality enhancement model can be correspondingly improved.

[0124] It should be noted that the multi-scale image quality enhancement model and the single-scale image quality enhancement model can be pre-trained offline, so in the actual image quality enhancement process, the multi-scale image quality enhancement model and the single-scale image quality enhancement model can be directly used in the enhancement method of this embodiment to improve the efficiency of image quality enhancement.

[0125] S3: Acquire a corresponding quality-enhanced video according to the quality-enhanced image frame set;

[0126] S4: Convert the image quality enhanced video into a multi-scale enhanced video with different resolutions and bit rates.

[0127] As described above, the image quality enhanced video obtained by the multi-scale image quality enhancement model contains image quality enhancement information at various resolutions and bit rates, so the multi-scale enhanced videos with different resolutions and bit rates obtained after the final conversion can still retain the enhanced details.

[0128] This embodiment uses the single-scale image quality enhancement model to achieve the best possible enhancement effect on each resolution scale at multiple different resolution scales and reasonably superimpose the enhancement effects, thereby obtaining a training enhanced image frame set for multi-scale image quality enhancement and training to obtain a multi-scale image quality enhancement model. Based on the multi-scale image quality enhancement model, this embodiment only needs to perform the enhancement operation once to achieve obvious image quality enhancement effects at multiple different resolution scales. Specifically, based on the trained multi-scale image quality enhancement model, when processing low-definition videos of different resolutions, the original low-definition video is first enhanced, and then through the conversion and adaptation of different resolutions and bit rates, it can reflect a significant image quality improvement effect at each resolution, thereby greatly saving computing resources.

[0129] Based on the same inventive concept, this embodiment also provides a video quality enhancement device, such as Figure 5 As shown, the enhancement device may include:

[0130] The video acquisition processing module 11 is used to acquire the source video and construct a corresponding source video image frame set according to the frame images of the source video;

[0131] In this embodiment, the video acquisition processing module 11 can be used to perform Figure 2 As shown in step S1, for the specific description of the video acquisition processing module 11, reference may be made to the description of step S1.

[0132] The image quality enhancement module 12 is used to input the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract an image quality enhanced image frame set;

[0133] In this embodiment, the image quality enhancement module 12 can be used to perform Figure 2 As shown in step S2, for the specific description of the image quality enhancement module 12, reference can be made to the description of step S2.

[0134] A video restoration module 13, configured to obtain a corresponding quality-enhanced video according to the quality-enhanced image frame set;

[0135] In this embodiment, the video restoration module 13 can be used to perform Figure 2 As shown in step S3, for the specific description of the video restoration module 13, reference may be made to the description of step S3.

[0136] The video conversion module 14 is used to convert the resolution and bit rate of the image quality enhanced video to obtain a plurality of multi-scale enhanced videos with different resolutions and different bit rates;

[0137] In this embodiment, the video conversion module 14 can be used to perform Figure 2As shown in step S4, for the detailed description of the video conversion module 14, reference may be made to the description of step S4.

[0138] Furthermore, the enhancement device may also include a multi-scale model training module 15, and the multi-scale model training module 15 is used to train the multi-scale image quality enhancement model;

[0139] In this embodiment, the multi-scale model training module 15 can be used to perform Figure 3 For the steps A1-A6 shown, the specific description of the multi-scale model training module 15 can refer to the description of the steps A1-A6.

[0140] Furthermore, the enhancement device may further include a single-scale model training module 16, and the single-scale model training module 16 is used to train the single-scale image quality enhancement model;

[0141] In this embodiment, the single-scale model training module 16 can be used to perform Figure 4 For the steps B1-B5 shown, the specific description of the single-scale model training module 16 can refer to the description of the steps B1-B5.

[0142] Based on the same inventive concept, this embodiment also provides an electronic device, Figure 6 The structural diagram of the electronic device of this embodiment is shown, which includes a memory 21 and a processor 22. The memory 21 stores computer-readable instructions, and the processor 22 executes the computer-readable instructions to implement the video quality enhancement method of this embodiment.

[0143] Preferably, the electronic device further includes a bus 23 and a communication interface 24 , and the processor 22 , the communication interface 24 and the memory 21 are connected via the bus 23 .

[0144] The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 24 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 23 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus 23 may be divided into an address bus, a data bus, a control bus, etc. (not fully drawn in the figure).

[0145] The processor 22 may be an integrated circuit chip with signal processing capability. In the specific implementation process, each step in the above method embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor 22, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the method of the above embodiment in combination with its hardware.

[0146] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor 22, the computer-executable instructions prompt the processor 22 to implement the above-mentioned video quality enhancement method. The specific implementation can be found in the embodiment, which will not be repeated here.

[0147] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0148] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for enhancing video quality, characterized in that: The enhancement method comprises: Acquire a source video, and construct a corresponding source video image frame set according to the frame images of the source video; Inputting the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract an image quality enhancement image frame set; the multi-scale image quality enhancement model is used to obtain the image quality enhancement image frame set of the source video at different resolutions and different bit rates through the source video image frame set; Acquire a corresponding quality-enhanced video according to the quality-enhanced image frame set; The image quality enhanced video is subjected to resolution and bit rate conversion to obtain a plurality of multi-scale enhanced videos with different resolutions and different bit rates.

2. A method for enhancing video quality according to claim 1, characterized in that: The training of the multi-scale image quality enhancement model specifically includes: Obtain a number of training video data with different resolutions and the same content; According to a number of time points, corresponding image frames are extracted from each of the training video data, and all the image frames extracted from each of the training video data are used to construct a training image frame set corresponding to the training video data; Inputting the plurality of training image frame sets into a pre-trained single-scale image quality enhancement model respectively to obtain corresponding training enhanced image frame sets; Converting all the acquired training enhanced image sets to the same target resolution and superimposing them to obtain a multi-scale training image frame set; the several resolutions include the target resolution; Inputting the training image frame set corresponding to the target resolution into the multi-scale image quality enhancement model to obtain a multi-scale predicted image frame set; The multiscale loss of the multiscale training image frame set and the multiscale prediction image frame set is calculated, and the multiscale image quality enhancement model is updated according to the multiscale loss.

3. A method for enhancing video quality according to claim 2, characterized in that: The step of inputting the plurality of training image frame sets into a pre-trained single-scale image quality enhancement model to obtain corresponding training enhanced image frame sets specifically includes: Extracting enhanced residual image sets corresponding to a number of training image frame sets respectively through the single-scale image quality enhancement model; The images in the enhanced residual image set are superimposed with the corresponding images in the training image frame set to obtain the training enhanced image frame set.

4. A method for enhancing video quality according to claim 3, characterized in that: The image in the enhanced residual image set is superimposed with the corresponding image in the training image frame set, which is expressed as follows: HR r =LR r +M(LR r )*ResidualFactor r In the formula, LR r represents the image in the training image frame set with resolution r, HR r represents the image in the training enhanced image frame set with a resolution of r obtained after superposition, M(LR r ) represents the LR extracted by the single-scale image quality enhancement model r The corresponding enhanced residual image, ResidualFactor r Represents the residual enhancement coefficient with resolution r.

5. The method for enhancing video quality according to claim 1, characterized in that: The training of the single-scale image quality enhancement model specifically includes: Obtain a number of high-definition training video data with different resolutions and the same content; Keeping the resolution unchanged, respectively reducing the bit rates of the high-definition training video data with different resolutions to obtain corresponding low-definition training video data; According to a number of time points, corresponding image frames are extracted from the high-definition training video data and the low-definition training video data, respectively, and a high-definition training image frame set and a low-definition training image frame set are constructed according to all the image frames extracted from the high-definition training video data and the low-definition training video data, respectively; Inputting the low-definition training image frame set into the single-scale image quality enhancement model to obtain a corresponding single-scale prediction image frame set; The enhancement loss of the single-scale predicted image frame set and the corresponding high-definition training image frame set is calculated, and the single-scale image quality enhancement model is updated according to the enhancement loss.

6. A method for enhancing video quality according to claim 5, characterized in that: The single-scale image quality enhancement models include several ones; each of the single-scale image quality enhancement models is trained for one resolution.

7. A method for enhancing video quality according to any one of claims 2 to 6, characterized in that: The single-scale image quality enhancement model includes a plurality of residual blocks; The processing flow of the single-scale image quality enhancement model specifically includes: Performing convolution processing on the input image to obtain convolution features of the input image; Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature; Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks; The residual extraction feature output by the last residual block is used as the residual image of the input image.

8. A video quality enhancement device, characterized in that: The enhancement device comprises: A video acquisition processing module, used to acquire a source video and construct a corresponding source video image frame set according to the frame images of the source video; The image quality enhancement module is used to input the source video image frame set into a pre-trained multi-scale image quality enhancement model to extract the image quality enhanced image frame set; the multi-scale image quality enhancement model is used to obtain the image quality enhanced image frame set according to the video image residuals of the source video at several different resolutions corresponding to the source video image frame set; A video restoration module, used for obtaining a corresponding quality-enhanced video according to the quality-enhanced image frame set; The video conversion module is used to convert the resolution and bit rate of the image quality enhanced video to obtain a plurality of multi-scale enhanced videos with different resolutions and different bit rates.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video quality enhancement method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer-readable program is stored thereon, and when the computer-readable program is executed, a video quality enhancement method as described in any one of claims 1 to 7 is implemented.

Images