Video image quality enhancement method and device, electronic equipment and storage medium

Abstract

The invention provides a video image quality enhancement method and apparatus, an electronic device and a storage medium. The enhancement method comprises the steps of obtaining source video data and constructing a corresponding source video image set; inputting the source video image set into a pre-trained multi-gear image quality enhancement model, and respectively obtaining image quality enhancement image sets corresponding to the source video data under different resolutions and different code rates through a plurality of output channels in the multi-gear image quality enhancement model; and obtaining an enhanced video with a corresponding resolution and a corresponding code rate according to each image quality enhanced image set. Compared with the prior art, the image quality enhancement image sets corresponding to the source video data under different resolutions and different code rates are obtained through the plurality of output channels set by the multi-gear image quality enhancement model, and only one-time image quality enhancement operation needs to be carried out on the source video image set of the source video. On the premise of reducing the computing resource consumption of the playing equipment, videos of multiple different gears can be quickly and effectively obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, device, electronic device and storage medium for enhancing the quality of a video image. Background Art

[0002] Nowadays, live video streaming is becoming more and more common in modern society, especially in the fields of social media, e-sports, and online education. In live video streaming, multiple resolutions and bit rates are usually set to meet the diverse needs of viewers and optimize the resource usage of service providers. Traditional multi-level live video streaming requires image quality enhancement for each level separately, which consumes a lot of computing resources. In order to obtain multiple different resolution levels, traditional downsampling algorithms are usually required to achieve resolution reduction, such as bilinear interpolation, bicubic interpolation, LANCZOS and other downsampling algorithms. However, if these traditional downsampling algorithms are only applied after deployment, they cannot be optimized and adjusted in combination with the model training process. Summary of the invention

[0003] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a video image quality enhancement method, device, electronic device and storage medium for effectively enhancing the image quality of live video broadcasts at multiple levels while reducing the computing resource consumption of the playback device.

[0004] According to a first aspect of the present application, a method for enhancing video image quality is provided, the method comprising:

[0005] Acquire source video data, and construct a corresponding source video image set according to frame images of the source video data;

[0006] Inputting the source video image set into a pre-trained multi-level image quality enhancement model, and obtaining the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model;

[0007] According to each of the image quality enhanced image sets, an enhanced video with a corresponding resolution and a corresponding bit rate is obtained.

[0008] Optionally, the multi-level image quality enhancement model includes:

[0009] A picture quality enhancement network, wherein the picture quality enhancement network comprises a plurality of output channels, and the output channels are divided into a plurality of single picture quality class output channels; the picture quality enhancement network is used to enhance the picture quality according to the input image set, and output a plurality of enhanced image sets having the same resolution and different picture qualities through the divided plurality of single picture quality class output channels; the different picture qualities correspond to the different bit rates respectively;

[0010] The sampling network is used to perform upsampling or downsampling according to the enhanced image sets output by the image quality enhancement network, and obtain several enhanced image sets with different resolutions as output.

[0011] Optionally, the training of the multi-level image quality enhancement model specifically includes:

[0012] Obtain several training video data with different resolutions and the same content respectively;

[0013] According to several time points in the training video data, corresponding image frames are extracted from several training video data respectively, and corresponding training image data sets are constructed according to all image frames extracted from each training video data respectively;

[0014] Obtaining a training enhanced image data set corresponding to each of the training image data sets respectively through a pre-trained single-level image quality enhancement model;

[0015] Selecting one of the training image data sets, inputting the selected training image data set into the multi-level image quality enhancement model to be trained, and obtaining a plurality of enhanced prediction image data sets; one enhanced prediction image data set corresponds to one training image data set;

[0016] The multi-level losses of each of the enhanced prediction image data sets and the corresponding training enhanced image data sets are calculated respectively, and the multi-level image quality enhancement model is updated according to the multi-level losses respectively.

[0017] Optionally, the training of the single-level image quality enhancement model specifically includes:

[0018] Acquire a plurality of high-definition training video data; any two of the high-definition training video data have different resolutions and / or different bit rates;

[0019] Keeping the resolution and bit rate unchanged, each of the high-definition training video data is subjected to image quality degradation processing to obtain corresponding low-definition training video data;

[0020] Extracting corresponding image frames from the high-definition training video data and the corresponding low-definition training video data according to a number of time points, and constructing a high-definition training image set and a low-definition training image set according to all the image frames extracted from the high-definition training video data and the low-definition training video data;

[0021] Inputting the low-definition training image sets of different resolutions into the single-level image quality enhancement model to obtain corresponding single-level prediction image sets;

[0022] The single-level loss of the single-level prediction image set and the high-definition prediction training image set is calculated, and the single-level image quality enhancement model is updated according to the single-level loss.

[0023] Optionally, the single-level image quality enhancement model includes several models; each of the single-level image quality enhancement models is trained for a combination of resolution and bit rate.

[0024] Optionally, the step of maintaining the resolution and the bit rate unchanged and performing image quality degradation processing on each of the high-definition training video data to obtain corresponding low-definition training video data specifically includes:

[0025] Keeping the resolution and bit rate of the high-definition training video data unchanged;

[0026] The high-definition training video data is transmitted to a preset playback end through a live stream according to the corresponding resolution and bit rate, and the playback video data transmitted to the playback end is collected and used as the low-definition training video data.

[0027] Optionally, the image quality enhancement network is constructed based on a super-resolution network.

[0028] According to a second aspect of the present application, a video image quality enhancement device is provided, the enhancement device comprising:

[0029] A source video acquisition module, used to acquire source video data and construct a corresponding source video image set according to frame images of the source video data;

[0030] A multi-level enhancement module, used for inputting the source video image set into a pre-trained multi-level image quality enhancement model, and obtaining the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model;

[0031] The video restoration output module is used to obtain an enhanced video with a corresponding resolution and a corresponding bit rate according to each of the image quality enhanced image sets.

[0032] According to a third aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video image quality enhancement method described in the first aspect above.

[0033] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer-readable program is stored. When the computer-readable program is executed, a video image quality enhancement method described in the first aspect is implemented.

[0034] According to any one of the aspects provided above, the present application provides a method, device, and electronic device set storage medium for enhancing the quality of a video image, wherein a pre-trained multi-level quality enhancement model is used to improve the quality of a source video image set, wherein the multi-level quality enhancement model is provided with a plurality of output channels, and the multi-level quality enhancement model can obtain the quality enhanced image set corresponding to the source video data at different resolutions and different bit rates according to the set output channels; therefore, by using the multi-level quality enhancement model, it is only necessary to perform a quality enhancement operation on the source video image set of the source video once, so that multiple videos of different levels can be obtained quickly and effectively;

[0035] Furthermore, the present application provides a method, device, electronic device and storage medium for enhancing video image quality. By improving the super-resolution network, the image quality enhancement network of the multi-level image quality enhancement model is constructed, and the mature and effective image quality enhancement capability of the super-resolution network is utilized to effectively achieve image quality enhancement, thereby improving the image quality enhancement capability of the constructed image quality enhancement network. On this basis, by adjusting the output channel of the image quality enhancement network, the original output channel is divided into a single image quality type output channel for outputting images of different image qualities, and it can be achieved through the single image quality type output channel, and a plurality of enhanced images of different image qualities can be obtained through a single image quality enhancement operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of an application scenario of the enhancement method provided in this embodiment.

[0037] Figure 2 This is a flowchart of the steps of the enhancement method provided in this embodiment.

[0038] Figure 3 A flowchart of the steps for training a multi-level image quality enhancement model provided in this embodiment.

[0039] Figure 4 A flowchart of the steps for training a single-level image quality enhancement model provided in this embodiment.

[0040] Figure 5 This is a structural diagram of the enhancement device provided in this embodiment.

[0041] Figure 6 The electronic device structure diagram provided for this embodiment is marked with: server 100, terminal 200, source video acquisition module 11, multi-speed enhancement module 12, video restoration output module 13, multi-speed model training module 14, single-speed model training module 15, memory 21, processor 22, bus 23, communication interface 24. DETAILED DESCRIPTION

[0042] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0043] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Nowadays, live video streaming is becoming more and more common in modern society, especially in areas such as social media, e-sports, and online education.

[0046] Among them, the network conditions of different viewers vary greatly. Some viewers may use high-speed networks such as high-speed broadband, high-speed Wi-Fi or 5G networks, which enables these viewers to watch higher-resolution, higher-quality and smoother live videos, while others may rely on low-speed networks such as 4G networks or public Wi-Fi. These viewers may pay more attention to the smoothness of live video. In addition to the influence of network factors, the viewing devices used by viewers will also affect the viewing experience of viewers. The viewing devices used by viewers range from large HD TV screens to small smartphone screens. Different display screens have different display effects and will also bring different viewing experiences to viewers.

[0047] Therefore, providing viewers with a variety of resolutions and bit rates in live video broadcasts can meet the diverse needs of viewers and optimize the resource usage of service providers. In traditional multi-level live video broadcasts, in order to reduce the amount of data transmission, in the process of transmitting live video data, in order to improve transmission efficiency, the live video is usually compressed. Bit rate compression can reduce data transmission without significantly reducing image quality, thereby improving the transmission efficiency of live video broadcasts at all levels. However, bit rate compression technology actually causes a reduction in image quality. Therefore, in order to ensure the image quality of live video broadcasts, it is necessary to enhance the image quality of the live video after transmission.

[0048] Because it involves live video broadcasts at multiple levels, traditional image quality enhancement technology needs to enhance the image quality for each level separately, which requires a lot of computing resources. In order to obtain multiple different resolution levels, it is usually necessary to use traditional downsampling algorithms to achieve resolution reduction, such as bilinear interpolation, bicubic interpolation, LANCZOS and other downsampling algorithms. However, if these traditional downsampling algorithms are only applied after deployment, they cannot be optimized and adjusted in combination with the model training process.

[0049] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0050] For example, a schematic diagram of an application scenario of a method for enhancing video image quality provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal 200 has a streaming media playback function and can also have an image processing function.

[0051] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0052] In one practicable manner, the server 100 and the terminal 200 may respectively execute a video image quality enhancement method provided in an embodiment of the present application, or, optionally, the video image quality enhancement method provided in an embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0053] like Figure 2 As shown, this embodiment provides a method for enhancing video image quality, and the enhancement method includes:

[0054] S1: Acquire source video data, and construct a corresponding source video image set according to frame images of the source video data;

[0055] In this embodiment, the source video data can be obtained based on actual live video or video clips. It can be understood that the enhancement method of this embodiment can be applied to image quality enhancement of various forms of videos and live broadcasts. In addition, the enhancement method can also be used to enhance the image quality of a single image or a small number of images. If the enhanced object is an image, an image set is directly constructed based on the image for enhancement.

[0056] In addition, it should be noted that since the purpose of this embodiment is to achieve enhancement of video image quality, this embodiment does not limit the means of collecting the source video data or other subsequent data that needs to be collected. The means of collecting various data can be set according to the corresponding data type or actual scenario.

[0057] At the same time, the specific collected data objects of this embodiment can also be adjusted according to the specific application scenarios of the enhancement method of this embodiment. It can be understood that, such as the above-mentioned live video, in the live video link, the video YUV frame is obtained after video decoding, and the processing object of the conventional image quality enhancement model is RGB image, so the training set of the image quality enhancement model can be constructed by correspondingly collecting video YUV frames, so that the model can learn to enhance the image quality of video YUV frames; for conventional video clips, the video content can be directly collected, and the video clips can be split into RGB image frames. According to the RGB image frames, the training set of the image quality enhancement model is constructed, so that the model can learn to enhance the image quality of RGB image frames.

[0058] S2: inputting the source video image set into a pre-trained multi-level image quality enhancement model, and obtaining the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model;

[0059] Specifically, in this embodiment, the multi-level image quality enhancement model may include at least an image quality enhancement network and a sampling network;

[0060] Specifically, in this embodiment, the image quality enhancement network can be set based on an existing super-resolution network. Preferably, the image quality enhancement network can be set based on a super-resolution ESRGAN (Enhanced Super-ResolutionGenerative Adversarial Networks). The super-resolution ESRGAN has a good image quality enhancement capability and can effectively achieve image quality enhancement of live video. It is understandable that the image quality enhancement network of this embodiment does not need to change the resolution of the input image, so the image quality enhancement network is constructed based on the image quality enhancement structure in the super-resolution network.

[0061] At the same time, the network structure of the super-resolution network in the prior art is complex, and the amount of calculation and parameters is very large. In order to meet the scene of real-time video image quality enhancement such as live video in this embodiment, this embodiment cuts the super-resolution network, and the specific cutting is as follows:

[0062] Reduce the number of residual blocks: Several residual blocks are set in the super-resolution network, and the residual blocks are used to capture the details and features of the image, but too many residual blocks will significantly increase the amount of calculation. Therefore, in this embodiment, the number of residual blocks in the super-resolution is cropped, wherein the specific number of cropping is set according to the scale of the multi-speed image quality enhancement model actually deployed; illustratively, taking the super-resolution ESRGAN as an example, the number of residual blocks in the existing super-resolution ESRGAN is 16, and by cropping the residual blocks, the number of residual blocks can be reduced to 6; by reducing the number of residual blocks, the consumption of computing resources can be reduced to a large extent, while retaining the details and feature information of the image as much as possible.

[0063] Reduce the number of channels: The super-resolution network contains several channels. The number of channels represents the number of filters used in each convolution operation, which directly affects the computational complexity of the model. Therefore, in this embodiment, the number of channels of the super-resolution network is cropped, and the specific number of cropping is set according to the scale of the multi-speed image quality enhancement model actually deployed; for example, taking the super-resolution ESRGAN as an example, the existing super-resolution ESRGAN is usually provided with 256 channels, and the number of channels can be reduced to 32 by cropping the channels, that is, reduced to 1 / 8 of the original number. Doing so can significantly reduce the demand for computing resources while retaining the necessary image detail information as much as possible.

[0064] The image quality enhancement network obtained after cropping based on the super-resolution network, although with a smaller scale, still has the ability to enhance image quality. However, in this embodiment, it is necessary to output multiple levels of image quality enhancement results through the multi-level image quality enhancement model, so the image quality enhancement network needs to be further improved.

[0065] The image quality enhancement network obtained based on the super-resolution network is the same as the super-resolution network and includes several output channels, but the conventional super-resolution network outputs a network of one resolution through all its output channels. For example, if the super-resolution network includes 3 output channels and the super-resolution network needs to output a 1080p image quality enhanced image, the 1080p image quality enhanced image is output through its 3 output channels; and it can output multiple levels of image quality enhancement results at the same time. In this embodiment, the output channels are divided into several single-image quality output channels according to the image quality type of the image to be output; in addition, in order to retain the output capacity of the output channels, the number of the output channels can be enhanced according to the image quality type of the image to be output. For example, if the image quality enhanced image is output with 3 output channels, and two image quality enhancement results need to be output in this embodiment, the number of output channels can be increased to 6 first, and then 3 of the output channels are divided into one single-image quality output channel to output one image quality enhanced image, and the remaining 3 are divided into another single-image quality output channel for outputting another image quality enhanced image.

[0066] Although the improved image quality enhancement image can output image quality enhancement images of different image quality types, in this embodiment, the image quality corresponds to the bit rate of the source video data of the input image, and the image quality enhancement network does not change the resolution of the input image, that is, through the image quality enhancement image, an image quality enhancement image with the same resolution and different image quality can be obtained.

[0067] In order to further obtain image quality enhanced images of different resolutions, the sampling network is set in the multi-level image quality enhancement model. The sampling network is used to upsample or downsample the enhanced image sets output by the image quality enhancement network to obtain several enhanced image sets of different resolutions as output.

[0068] Specifically, the upsampling and downsampling of the sampling network can be set according to the resolution of the input image and the resolution of different levels that need to be output. For example, if the resolution of the image quality enhancement image output by the image quality enhancement network is 1080p (including images of different image qualities), and it is necessary to output three image quality enhancement images with resolutions of 1080p, 720p and 540p, then the 1080p image quality enhancement image is downsampled to obtain image quality enhancement images of 720p and 540p respectively; if the resolution of the output image quality enhancement image is 720p, and it is necessary to output three image quality enhancement images with resolutions of 1080p, 720p and 540p, then the 720p image quality enhancement image is upsampled to obtain a 1080p image quality enhancement image, and downsampled to obtain a 540p image quality enhancement image.

[0069] The improved multi-level image quality enhancement model can effectively output image quality enhancement results of multiple levels. In order to achieve the enhancement capability of the multi-level image quality enhancement model, it is necessary to conduct targeted training on the multi-level enhancement model, such as Figure 3 As shown, the training may specifically include:

[0070] A1: Obtain several training video data with different resolutions and the same content;

[0071] As we all know, there are differences in image quality details between videos of different resolutions. For example, high-quality image quality details at 1080P resolution may not be fully retained after conversion to 720P and 540P resolutions, and vice versa. These differences are not only reflected in video acquisition and encoding, but also affect the way of image quality enhancement. In fact, as mentioned above, the image quality enhancement method applicable to 1080P resolution may lose a large degree of enhanced image quality details when the resolution is reduced to 720P or 540P. Similarly, when the image quality enhancement methods of 720P and 540P are applied to 1080P resolution, they may not be fine enough due to too many details, resulting in unclear enhancement effects or noise.

[0072] In order to avoid excessive consumption of computing resources and cover multiple different resolutions and bit rate levels at the same time, the multi-level image quality enhancement model in this embodiment can effectively achieve significant image quality improvement at different resolutions. In order to effectively train the multi-level image quality enhancement model so that the multi-level image quality enhancement model can better adapt to various different resolutions, it is necessary to collect several training video data with different resolutions and the same content. Specifically, live video streams with different resolutions can be collected as the training video data, wherein the collected training video data is preferably low-definition video data with lower picture quality, so as to better realize the enhanced learning of the multi-level image quality enhancement model; in addition, the collection of the training video data can also be based on collecting low-definition video data of one resolution, and converting the resolution of the basic low-definition video data to obtain low-definition video data of other resolutions.

[0073] A2: extracting corresponding image frames from a plurality of the training video data respectively according to a plurality of time points in the training video data, and constructing a corresponding training image data set respectively according to all the image frames extracted from each training video data;

[0074] Since the contents of several of the training video data are the same, the corresponding frames at the same time point are the same. By extracting frames at the same time point in each of the training video data, corresponding image frames with different resolutions in several of the training video data can be obtained, and then all the image frames extracted from each of the training video data are constructed into a corresponding training image data set. This can ensure that in the subsequent learning process, the multi-level image quality enhancement model can learn the image details of images with the same content at different resolutions, and then can fully express the image details of different resolutions in the predicted image in the subsequent prediction process.

[0075] During the acquisition process, it is necessary to adjust the frame extraction frequency and determine the frame extraction time point to ensure that the extracted image frames contain different scenes as much as possible, such as indoors and outdoors, day and night, static and dynamic environments, etc., to ensure the diversity and comprehensiveness of the data.

[0076] A3: Obtaining a training enhanced image data set corresponding to each training image data set respectively through a pre-trained single-level image quality enhancement model;

[0077] In this embodiment, the multi-level image quality enhancement model needs to be supervised by reference to image quality enhancement images. In this embodiment, since the multi-level image quality enhancement model outputs image quality enhancement images of multiple levels, supervised learning is required for the image quality enhancement images of each level, wherein the image quality enhancement images of each level can be obtained through the pre-trained single-level image quality enhancement model.

[0078] Preferably, the single-level image quality enhancement model can be constructed based on the existing super-resolution network and trained accordingly, specifically, as follows: Figure 4 As shown, the training of the single-level image quality enhancement model may include:

[0079] B1: Obtain some high-definition training video data;

[0080] In this embodiment, in order to effectively achieve image quality enhancement at multiple resolutions and bitrate levels, preferably, any two of the high-definition training video data have different resolutions and / or different bitrates;

[0081] Specifically, the acquisition of the high-definition training video data can be performed by directly collecting high-definition video clips, collecting high-definition video live broadcasts, and shooting with high-definition equipment, and by setting specific parameters such as resolution and bit rate, the high-definition training video data with different resolutions and / or different bit rates can be obtained. Preferably, the acquisition of the high-definition training video data is performed by shooting with high-definition equipment, so that the high-definition training video data obtained does not undergo data compression during network transmission, and can well retain high-definition image details.

[0082] B2: Keeping the resolution and bit rate unchanged, respectively perform image quality degradation processing on each of the high-definition training video data to obtain corresponding low-definition training video data;

[0083] As described above, the video is compressed during transmission, which will reduce the quality of the video. Therefore, in this embodiment, the video data with degraded quality in a real environment can be obtained by simulating the compression of the video transmission process.

[0084] In this embodiment, the image quality degradation processing may specifically include:

[0085] Keeping the resolution and bit rate of the high-definition training video data unchanged;

[0086] The high-definition training video data is transmitted to a preset playback end through a live stream according to the corresponding resolution and bit rate, and the playback video data transmitted to the playback end is collected and used as the low-definition training video data.

[0087] Exemplarily, if you need to obtain the low-definition training video data with a resolution of 1080p and a bit rate of 10m, then the high-definition training video data with a resolution of 1080p and a bit rate of 10m is transmitted to the playback end through a live stream, and the live resolution is set to 1080p and the live bit rate is set to 10m. The high-definition training video data is played after being transmitted to the playback end. At this time, the video played is the low-definition training video data with a resolution of 1080p and a bit rate of 10m. The low-definition training video data of other gears can be obtained accordingly according to the above method. By obtaining the video played by the live stream, the complete live link degradation process can be simulated more realistically, which helps the model to better learn and cope with degradation conditions in the actual environment. In this way, it can provide an important data basis for subsequent research work such as video quality improvement and compression algorithm optimization.

[0088] B3: extracting corresponding image frames from the high-definition training video data and the corresponding low-definition training video data according to a number of time points, and constructing a high-definition training image set and a low-definition training image set according to all the image frames extracted from the high-definition training video data and the low-definition training video data;

[0089] Similar to step A2, by extracting frames from the high-definition training video data and the low-definition training video data respectively, the corresponding high-definition training image set and low-definition training image set are obtained. The high-definition training image set can be used as a reference to effectively realize supervised learning of the model.

[0090] Similarly, the frame extraction of the high-definition training video data and the low-definition training video data also needs to adjust the frame extraction frequency and determine the frame extraction time point to ensure that the extracted image frames contain as many different scenes as possible to ensure the diversity and comprehensiveness of the data.

[0091] B4: inputting the low-definition training image sets of different resolutions into the single-level image quality enhancement model to obtain corresponding single-level prediction image sets;

[0092] B5: Calculate the single-level loss of the single-level prediction image set and the high-definition prediction training image set, and update the single-level image quality enhancement model according to the single-level loss.

[0093] Specifically, before the single-level image quality enhancement model has completed training, its output result has not converged, that is, the image quality enhancement has not been achieved, so it is necessary to provide an enhanced "target" for the output of the single-level image quality enhancement model so that the model converges to this "target" during the training process;

[0094] In this embodiment, since the low-definition training image set input into the single-level image quality enhancement model is obtained by degrading the image quality of the corresponding high-definition image quality enhancement model, the high-definition training image set corresponding to the low-definition training image set can well serve as the "target" of the single-level image quality enhancement model. By calculating the single-level loss of the high-definition prediction image set and the corresponding high-definition training image set, the single-level image quality enhancement model can better learn image quality enhancement, and finally obtain the single-level image quality enhancement model that can output enhanced detail information with different resolutions.

[0095] It can be understood that in a preferred implementation of the present embodiment, the single-level image quality enhancement model can learn and predict video data of multiple different resolutions and / or bit rates. For example, video data of multiple different resolutions and / or bit rates can be input into a single-level image quality enhancement model for learning, and the obtained single-level image quality enhancement model can predict video data of multiple different resolutions and / or bit rates. In this implementation, the single-level image quality enhancement model has higher versatility, and there is no need to distinguish between video data of different resolutions, and prediction can be directly performed through the single-level image quality enhancement model.

[0096] In another preferred implementation of the present embodiment, multiple single-level image quality enhancement models can be set, and the setting data of the single-level image quality enhancement model matches the type of resolution and bit rate of the video data to be predicted, that is, one single-level image quality enhancement model only predicts video data of one resolution and bit rate combination. In this implementation, the single-level image quality enhancement model is targeted and can more accurately predict the video data it targets, thereby effectively improving the accuracy of the prediction.

[0097] At the same time, the above two implementation modes can be combined, that is, the same single-level image quality enhancement model can be used to predict video data within a certain resolution range and bit rate range, and a single-level image quality enhancement model can be used to predict video data with a certain specific resolution and bit rate. The specific settings can be made according to the actual application scenario.

[0098] Furthermore, the single-level loss function may adopt mean square error, peak signal-to-noise ratio, structural similarity index, etc., which may be specifically set according to actual needs and will not be further limited here.

[0099] A4: selecting one of the training image data sets, inputting the selected training image data set into the multi-level image quality enhancement model to be trained, and obtaining a plurality of enhanced prediction image data sets;

[0100] Wherein, one of the enhanced prediction image datasets corresponds to one of the training image datasets;

[0101] Specifically, since the image quality enhancement network in the multi-level image quality enhancement model can output image quality enhancement images with the same resolution and different image qualities, and the sampling network in the multi-level image quality enhancement model can obtain image quality enhancement images with the same resolution through upsampling and downsampling, one of the training image data sets obtained in step A3 can be selected as the training set input of the multi-level image quality enhancement model in each training cycle. The specific selection method can be random selection, or selection of different resolutions and bitrate levels in rotation, to ensure that the multi-level image quality enhancement model can cover all levels during the training process, thereby improving the image quality enhancement capability of the multi-level image quality enhancement model.

[0102] A5: Calculate the multi-level losses of each enhanced prediction image data set and the corresponding training enhanced image data set respectively, and update the multi-level image quality enhancement model according to the multi-level losses respectively.

[0103] Since the multi-level image quality enhancement model has not completed training, the obtained multi-level predicted image frame set cannot achieve a good enhancement effect. Similar to the single-level image quality enhancement model, it is necessary to provide the multi-level image quality enhancement model with high-quality images at corresponding resolutions and bit rates as a "target". Through supervised learning, the multi-level image quality enhancement model converges to this "target" during training.

[0104] Each of the enhanced predicted image data sets obtained in step A4 corresponds to a training image data set, that is, it can correspond to a training enhanced image data set output by the single-level image quality enhancement model, and the training enhanced image data set is obtained after being enhanced by the single-level image quality enhancement model. Therefore, the training enhanced image data set can well serve as the "target" of the enhanced predicted image data set corresponding to the output of the multi-level image quality enhancement model, so that the multi-level image quality enhancement model can fully learn the enhancement of each resolution and bit rate during the learning process, and finally make each of the enhanced predicted image data sets output by the multi-level image quality enhancement model include resolutions and bit rates covering various levels and achieve image quality enhancement.

[0105] Specifically, in this embodiment, the loss function of the multi-level loss may adopt mean square error, peak signal-to-noise ratio, structural similarity index, etc., which may be set according to actual needs and is not further limited here.

[0106] S3: Obtain enhanced videos with corresponding resolutions and corresponding bit rates according to each of the image quality enhanced image sets.

[0107] It can be understood that in the present embodiment, the above-mentioned multi-level image quality enhancement model and single-level image quality enhancement model can be obtained by pre-training offline, so in the actual image quality enhancement process of the present embodiment, the pre-trained multi-level image quality enhancement model and single-level image quality enhancement model can be directly used in the enhancement method of the present embodiment. In the process of image quality enhancement, it is only necessary to obtain the source video data and perform one image quality enhancement operation to obtain multiple enhanced videos of different levels through the multi-level image quality enhancement model. No additional equipment computing resources are required, which can greatly improve the efficiency of image quality enhancement.

[0108] Based on the same inventive concept, this embodiment also provides a video image quality enhancement device, such as Figure 5 As shown, the enhancement device may include:

[0109] A source video acquisition module 11 is used to acquire source video data and construct a corresponding source video image set according to frame images of the source video data;

[0110] In this embodiment, the source video acquisition module 11 can be used to perform Figure 2 As shown in step S1, for the specific description of the source video acquisition module 11, reference may be made to the description of step S1.

[0111] The multi-level enhancement module 12 is used to input the source video image set into a pre-trained multi-level image quality enhancement model, and obtain the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model;

[0112] In this embodiment, the multi-gear enhancement module 12 can be used to perform Figure 2 As shown in step S2, for the detailed description of the multi-gear enhancement module 12, reference may be made to the description of step S2.

[0113] The video restoration output module 13 is used to obtain an enhanced video with a corresponding resolution and a corresponding bit rate according to each of the image quality enhanced image sets.

[0114] In this embodiment, the video restoration output module 13 can be used to perform Figure 2 As shown in step S2, for the specific description of the video restoration output module 13, reference may be made to the description of step S2.

[0115] A multi-level model training module 14, used for training the multi-level image quality enhancement model;

[0116] In this embodiment, the multi-gear model training module 14 can be used to perform Figure 3 For the detailed description of the multi-gear model training module 14 , please refer to the description of the steps A1-A5.

[0117] A single-level model training module 15, used for training the single-level image quality enhancement model;

[0118] In this embodiment, the single gear model training module 15 can be used to perform Figure 4 For the detailed description of the single-gear model training module 15 , reference can be made to the description of the steps B1-B5.

[0119] This embodiment also provides an electronic device, Figure 6 The structural diagram of the electronic device of this embodiment is shown, which includes a memory 21 and a processor 22. The memory 21 stores computer-readable instructions, and the processor 22 executes the computer-readable instructions to implement the video image quality enhancement method of this embodiment.

[0120] Preferably, the electronic device further includes a bus 23 and a communication interface 24 , and the processor 22 , the communication interface 24 and the memory 21 are connected via the bus 23 .

[0121] The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 24 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 23 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus 23 may be divided into an address bus, a data bus, a control bus, etc. (not fully drawn in the figure).

[0122] The processor 22 may be an integrated circuit chip with signal processing capability. In the specific implementation process, each step in the above method embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor 22, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the method of the above embodiment in combination with its hardware.

[0123] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor 22, the computer-executable instructions prompt the processor 22 to implement the above-mentioned video image quality enhancement method. The specific implementation can be found in the embodiment, which will not be repeated here.

[0124] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0125] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for enhancing video image quality, characterized in that: The enhancement method comprises: Acquire source video data, and construct a corresponding source video image set according to frame images of the source video data; Inputting the source video image set into a pre-trained multi-level image quality enhancement model, and obtaining the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model; According to each of the image quality enhanced image sets, an enhanced video with a corresponding resolution and a corresponding bit rate is obtained.

2. A method for enhancing video image quality according to claim 1, characterized in that: The multi-level image quality enhancement model includes: A picture quality enhancement network, wherein the picture quality enhancement network comprises a plurality of output channels, and the output channels are divided into a plurality of single picture quality class output channels; the picture quality enhancement network is used to enhance the picture quality according to the input image set, and output a plurality of enhanced image sets having the same resolution and different picture qualities through the divided plurality of single picture quality class output channels; the different picture qualities correspond to the different bit rates respectively; The sampling network is used to perform upsampling or downsampling according to the enhanced image sets output by the image quality enhancement network, and obtain several enhanced image sets with different resolutions as output.

3. A method for enhancing video image quality according to any one of claims 1 or 2, characterized in that: The training of the multi-level image quality enhancement model specifically includes: Obtain several training video data with different resolutions and the same content respectively; According to several time points in the training video data, corresponding image frames are extracted from several training video data respectively, and corresponding training image data sets are constructed according to all image frames extracted from each training video data respectively; Obtaining a training enhanced image data set corresponding to each of the training image data sets respectively through a pre-trained single-level image quality enhancement model; Selecting one of the training image data sets, inputting the selected training image data set into the multi-level image quality enhancement model to be trained, and obtaining a plurality of enhanced prediction image data sets; one enhanced prediction image data set corresponds to one training image data set; The multi-level losses of each of the enhanced prediction image data sets and the corresponding training enhanced image data sets are calculated respectively, and the multi-level image quality enhancement model is updated according to the multi-level losses respectively.

4. A method for enhancing video image quality according to claim 3, characterized in that: The training of the single-level image quality enhancement model specifically includes: Acquire a plurality of high-definition training video data; any two of the high-definition training video data have different resolutions and / or different bit rates; Keeping the resolution and bit rate unchanged, each of the high-definition training video data is subjected to image quality degradation processing to obtain corresponding low-definition training video data; Extracting corresponding image frames from the high-definition training video data and the corresponding low-definition training video data according to a number of time points, and constructing a high-definition training image set and a low-definition training image set according to all the image frames extracted from the high-definition training video data and the low-definition training video data; Inputting the low-definition training image sets of different resolutions into the single-level image quality enhancement model to obtain corresponding single-level prediction image sets; The single-level loss of the single-level prediction image set and the high-definition prediction training image set is calculated, and the single-level image quality enhancement model is updated according to the single-level loss.

5. A method for enhancing video image quality according to claim 4, characterized in that: The single-level image quality enhancement models include several ones; each of the single-level image quality enhancement models is trained for a combination of a resolution and a bit rate.

6. A method for enhancing video image quality according to claim 4, characterized in that: The step of maintaining the resolution and the bit rate unchanged and performing image quality degradation processing on each of the high-definition training video data to obtain corresponding low-definition training video data specifically includes: Keeping the resolution and bit rate of the high-definition training video data unchanged; The high-definition training video data is transmitted to a preset playback end through a live stream according to the corresponding resolution and bit rate, and the playback video data transmitted to the playback end is collected and used as the low-definition training video data.

7. A method for enhancing video image quality according to any one of claims 1 or 2, characterized in that: The image quality enhancement network is constructed based on a super-resolution network.

8. A video image quality enhancement device, characterized in that: The enhancement device comprises: A source video acquisition module, used to acquire source video data and construct a corresponding source video image set according to frame images of the source video data; A multi-level enhancement module, used for inputting the source video image set into a pre-trained multi-level image quality enhancement model, and obtaining the image quality enhancement image sets corresponding to the source video data at different resolutions and different bit rates through several output channels of the multi-level image quality enhancement model; The video restoration output module is used to obtain an enhanced video with a corresponding resolution and a corresponding bit rate according to each of the image quality enhanced image sets.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video image quality enhancement method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer-readable program is stored thereon, and when the computer-readable program is executed, a video image quality enhancement method as described in any one of claims 1 to 7 is implemented.

Images