Video image quality enhancement method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video image quality enhancement method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. According to the method, the training of overlapping the image quality enhancement effect for multiple times is performed on the high-definition video, the enhanced high-definition image set with the improved image quality is finally obtained, the image quality enhancement model is trained by using the enhanced high-definition image set, and the image quality enhancement capability of the image quality enhancement model is effectively improved. The image quality of the original video frame in the live video is enhanced through the image quality enhancement model to obtain the enhanced video frame of which the image quality is obviously improved, and the definition of the live picture can be ensured under the condition of reducing the live code rate due to the fact that the image quality of the enhanced video frame is obviously improved. The code rate redundancy can be effectively reduced by reducing the code rate, and the bandwidth cost is remarkably saved, so that the network bandwidth requirement when audiences watch the ultra-high-definition live broadcast is reduced, and the audiences can watch the ultra-high-definition live broadcast more smoothly.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video quality enhancement method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of the live broadcast industry, the audience's requirements for live broadcast quality are also constantly increasing. Although the resolution of live broadcast technology and equipment is constantly improving, many classic popular images or videos played in live broadcasts were produced many years ago. The quality of these images or videos is relatively low. After using image quality enhancement technologies such as denoising, deblurring, sharpening, and super-resolution to enhance the image quality, the image quality still cannot meet the audience's pursuit of ultra-high-definition quality. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a video quality enhancement method, device, electronic device and storage medium, which can enhance the quality of live video and significantly save bandwidth costs.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, the present invention provides a method for enhancing video quality, the method comprising:

[0006] Decode the live video to obtain multiple original video frames;

[0007] Inputting the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is trained using a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of video frames in the high-definition video;

[0008] The enhanced video frame is encoded to obtain an enhanced video.

[0009] In an optional implementation manner, the image quality enhancement model is trained in the following manner:

[0010] Determine a low-definition training image according to the low-definition image set; the low-definition training image is a low-definition image in the low-definition image set, or an image obtained by downsampling the low-definition image in the low-definition image set;

[0011] Inputting the low-definition training image into an initial image super-resolution network to obtain a corresponding enhanced training image; the initial image super-resolution network is obtained by trimming the number of basic residual blocks and the number of channels of an open source image super-resolution network;

[0012] Based on the loss information of the enhanced training image and the corresponding image in the enhanced high-definition image set, the parameters of the initial image super-resolution network are iteratively updated to obtain the image quality enhancement model.

[0013] In an optional implementation manner, the enhanced high-definition image set is obtained by:

[0014] Acquire multiple high-definition videos and a to-be-trained image super-resolution network; the to-be-trained image super-resolution network includes a global residual connection;

[0015] According to each high-definition video, a plurality of high-definition video frames, the low-definition image set and the high-definition image set are obtained; the low-definition image set includes low-definition images with different bit rates;

[0016] Using the image super-resolution network to be trained as a target network and the high-definition image set as a target image set;

[0017] Obtaining a training residual according to the randomly acquired low-definition image and the target network;

[0018] Obtaining a target enhanced image according to the training residual, the preset residual coefficient and the corresponding low-definition image;

[0019] Iteratively updating the parameters of the target network according to the loss information between the target enhanced image and the corresponding image in the target image set to obtain a trained network;

[0020] Obtaining a new high-definition image set according to the trained network and each of the high-definition video frames;

[0021] When the image quality enhancement effect of the trained network meets the preset conditions, the new high-definition image set is used as the enhanced high-definition image set;

[0022] When the image quality enhancement effect of the trained network does not meet the preset conditions, the trained network is used as the target network, the new high-definition image set is used as the target image set, and the trained network is iteratively trained.

[0023] In an optional implementation manner, obtaining a plurality of high-definition video frames, the low-definition image set and the high-definition image set according to each high-definition video includes:

[0024] Encoding each of the high-definition videos according to different training bit rates to obtain multiple low-definition videos of each of the high-definition videos;

[0025] Extract frames of each of the high-definition videos and the multiple low-definition videos corresponding to the high-definition videos according to a preset frame extraction interval, and obtain multiple high-definition video frames corresponding to the high-definition videos and multiple low-definition images corresponding to each low-definition video;

[0026] Obtaining the low-definition image set according to all the low-definition images;

[0027] Each of the high-definition video frames is sharpened to obtain a sharpened image corresponding to each of the high-definition video frames, and the high-definition image set is obtained based on all of the sharpened images.

[0028] In an optional implementation, obtaining a training residual according to the randomly acquired low-definition image and the target network includes:

[0029] Adding random noise and / or performing random blur processing on the randomly acquired low-definition image to obtain a degraded image;

[0030] The degraded image is input into the target network to obtain a training residual.

[0031] In an optional implementation manner, obtaining a target enhanced image according to the training residual, a preset residual coefficient and the corresponding low-definition image includes:

[0032] Multiplying the training residual by the preset residual coefficient to obtain a residual vector;

[0033] The residual vector and the feature vector of the low-definition image are summed to obtain a target enhancement vector, and the target enhanced image is generated according to the target enhancement vector.

[0034] In an optional implementation manner, obtaining a new high-definition image set according to the trained network and each of the high-definition video frames includes:

[0035] Inputting each of the high-definition video frames into the trained network to obtain a target residual corresponding to the high-definition video frame;

[0036] Obtaining a high-definition image according to each of the target residuals, the preset residual coefficients and the corresponding high-definition video frames;

[0037] A new high-definition image set is obtained based on all the high-definition images.

[0038] In a second aspect, the present invention provides a video quality enhancement device, the device comprising:

[0039] A decoding module, used to decode the live video to obtain multiple original video frames;

[0040] An enhancement module is used to input the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is trained using a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of video frames in the high-definition video;

[0041] The encoding module is used to encode the enhanced video frame to obtain an enhanced video.

[0042] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the video quality enhancement method described in any of the aforementioned embodiments.

[0043] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a video quality enhancement method as described in any one of the aforementioned embodiments.

[0044] Compared with the prior art, the video quality enhancement method, device, electronic device and storage medium provided in the embodiments of the present invention perform multiple trainings of superimposed quality enhancement effects on high-definition videos, and finally obtain an enhanced version of a high-definition image set with improved quality, and use the enhanced version of the high-definition image set to train a quality enhancement model, thereby effectively improving the quality enhancement capability of the quality enhancement model. The quality of the original video frames in the live video is enhanced by the quality enhancement model to obtain enhanced video frames with significantly improved quality. Since the quality of the enhanced video frames is significantly improved, the clarity of the live picture can be guaranteed even when the live bit rate is reduced. Since reducing the bit rate can effectively reduce bit rate redundancy and significantly save bandwidth costs, the network bandwidth requirements for viewers watching ultra-high-definition live broadcasts are reduced, allowing viewers to watch ultra-high-definition live broadcasts more smoothly, significantly improving user experience.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of a process of a video quality enhancement method provided by an embodiment of the present invention is shown.

[0048] Figure 2 The figure shows the comparison of live broadcast images before and after image quality enhancement.

[0049] Figure 3Another schematic flow chart of a method for enhancing video quality provided by an embodiment of the present invention is shown.

[0050] Figure 4 Another schematic flow chart of a method for enhancing video quality provided by an embodiment of the present invention is shown.

[0051] Figure 5 A block diagram of a video quality enhancement device provided by an embodiment of the present invention is shown.

[0052] Figure 6 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0053] Icons: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication module; 400 - video quality enhancement device; 401 - decoding module; 402 - enhancement module; 403 - encoding module. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0056] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] Image quality enhancement refers to improving the visual quality of images through various technical means, significantly enhancing the clarity and details of images, so that users can get a better visual experience when watching images or videos. The technical means of image quality enhancement include denoising, deblurring, sharpening and super-resolution. These technologies each have their own unique functions and application scenarios:

[0058] 1) Denoising: Noise is a big enemy of image quality, especially in images taken under low light conditions. Denoising technology can effectively reduce the noise in the image, improve the signal-to-noise ratio of the image, and make the image look cleaner and clearer.

[0059] 2) Deblurring: Image blur may be caused by many reasons, such as motion during shooting, inaccurate focus, etc. Deblurring technology can restore the details in the image through a series of algorithms, making the blurred image clear again.

[0060] 3) Sharpening: Sharpening technology makes images look sharper and more vivid by enhancing the edges and details of the image. This is particularly critical for important details in the image, especially in scenes that require high precision.

[0061] 4) Super-resolution: Super-resolution technology can convert low-resolution images into high-resolution images, making the image more detailed. This is of great significance for application scenarios that require enlarged images, such as surveillance videos and satellite images.

[0062] The inventors have found that these technologies can be used to restore low-quality images affected by noise, blur, etc., making them clear and usable again. Image quality enhancement technology is widely used in the field of live broadcasting. During live broadcasting, the quality of the image directly affects the audience's viewing experience. Through real-time image quality enhancement technology, it can be ensured that the live video can maintain high clarity and details even under poor network conditions, so that the audience can have a better viewing experience. In addition, image quality enhancement can also keep the image of high quality after compression, thereby saving storage space and transmission bandwidth.

[0063] With the rapid development of the live broadcast industry, the audience's requirements for live broadcast quality are also constantly increasing. Nowadays, the resolution of live broadcast videos has basically reached 1080P, and many live broadcasts have even begun to adopt 2K and 4K resolutions. At the same time, more and more live broadcast users are also using 2K or 4K monitors, and even many flagship mobile devices have gradually popularized 2K resolution. This trend shows that the audience's demand for high resolution is becoming stronger and stronger, and live broadcast technology and hardware support are also constantly improving to meet this demand. However, despite the continuous improvement of the resolution of live broadcast technology and equipment, many classic popular images or videos were produced many years ago. The image quality of these images or videos is relatively low, which is difficult to meet the audience's pursuit of ultra-high-definition quality.

[0064] Based on this, the video quality enhancement method and device provided by the embodiment of the present invention perform multiple trainings of superimposed quality enhancement effects on high-definition videos, and finally obtain an enhanced version of a high-definition image set with improved quality, and use the enhanced version of the high-definition image set to train a quality enhancement model, thereby effectively improving the quality enhancement capability of the quality enhancement model. The quality of the original video frames in the live video is enhanced by the quality enhancement model to obtain enhanced video frames with significantly improved quality. Since the quality of the enhanced video frames is significantly improved, the clarity of the live picture can be guaranteed even when the live bit rate is reduced. Since reducing the bit rate can effectively reduce bit rate redundancy and significantly save bandwidth costs, the network bandwidth requirements for viewers watching ultra-high-definition live broadcasts are reduced, allowing viewers to watch ultra-high-definition live broadcasts more smoothly, significantly improving user experience.

[0065] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0066] Please refer to Figure 1 , Figure 1 A schematic diagram of a process flow of a video quality enhancement method provided by an embodiment of the present invention is shown. The method comprises the following steps:

[0067] Step S100: decode the live video to obtain multiple original video frames.

[0068] Step S110, input the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is trained using a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of the video frames in the high-definition video.

[0069] In an embodiment of the present invention, a live video is decoded to obtain a plurality of original video frames corresponding to the live video. The original video frames are sequentially input into a pre-trained image quality enhancement model, and image quality enhancement processing is performed to obtain an enhanced video frame corresponding to each original video frame.

[0070] Among them, the enhanced HD image set is obtained by iteratively enhancing the image quality of each video frame in the HD video. Compared with each video frame in the HD video, the enhanced HD image set has a higher resolution. The image quality enhancement model is obtained by iterative training using low-definition images and enhanced HD images corresponding to the same HD video. By continuously learning the image quality features of the enhanced HD images, the image quality enhancement effect of the image quality enhancement model is effectively improved. Figure 2 For example, Figure 2 In the live game broadcast, the original video frame is blurry, while the enhanced video frame has a clearer texture.

[0071] Step S120, encoding the enhanced video frame to obtain an enhanced video.

[0072] In the embodiment of the present invention, it is assumed that the original video frame is not enhanced. Since the quality of the original video frame is average without enhancement, in order for the audience to see a clearer picture, it is necessary to retain the details of the original video frame as much as possible. Therefore, it is necessary to use a higher bit rate to encode the original video frame, the video compression ratio is relatively small, and the network bandwidth occupied during the live broadcast is relatively large. That is, the higher the bit rate, the smaller the video compression ratio after encoding, and the larger the network bandwidth occupied.

[0073] After the original video frames are enhanced, enhanced video frames with higher picture quality are obtained. While ensuring that the audience can see clear pictures, due to the higher picture quality of the enhanced video frames, high-quality picture effects can be maintained when transmitted at a lower bit rate, which can effectively save network bandwidth costs.

[0074] In summary, the video quality enhancement method provided by the embodiment of the present invention performs multiple trainings of superimposed quality enhancement effects on high-definition videos, and finally obtains an enhanced version of a high-definition image set with improved quality, and uses the enhanced version of the high-definition image set to train a quality enhancement model, thereby effectively improving the quality enhancement capability of the quality enhancement model. The quality of the original video frames in the live video is enhanced by the quality enhancement model to obtain enhanced video frames with significantly improved quality. Since the quality of the enhanced video frames is significantly improved, the clarity of the live picture can be guaranteed even when the live bit rate is reduced. Since reducing the bit rate can effectively reduce bit rate redundancy and significantly save bandwidth costs, the network bandwidth requirements for viewers watching ultra-high-definition live broadcasts are reduced, allowing viewers to watch ultra-high-definition live broadcasts more smoothly, significantly improving user experience.

[0075] Optionally, before using the image quality enhancement model to enhance the live broadcast quality, the image quality enhancement model needs to be pre-trained. The following is an explanation of the training process of the image quality enhancement model. Figure 3 , Figure 3 Another schematic diagram of a process of enhancing the video quality provided by an embodiment of the present invention is shown. The video quality enhancement model is trained in the following manner:

[0076] Step S200, determining a low-definition training image according to the low-definition image set; the low-definition training image is a low-definition image in the low-definition image set, or an image obtained by downsampling the low-definition image in the low-definition image set.

[0077] In an embodiment of the present invention, when super-resolution is set, the low-definition images in the low-definition image set are downsampled to obtain low-definition training images. When super-resolution is not set, the low-definition images in the low-definition image set are used as low-definition training images. Common downsampling methods include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos resampling, etc. Assuming that 2x super-resolution is set, the low-definition image needs to be downsampled to 1 / 2 of the original to obtain a low-definition training image.

[0078] Step S210, inputting the low-definition training image into the initial image super-resolution network to obtain the corresponding enhanced training image; the initial image super-resolution network is obtained by cropping the number of basic residual blocks and the number of channels of the open source image super-resolution network.

[0079] In an embodiment of the present invention, in order to better adapt to the real-time live broadcast scene, the number of basic residual blocks and the number of channels of the open source image super-resolution network are cropped to obtain an initial image super-resolution network. The open source image super-resolution network can be a super-resolution generative adversarial network (Super-Resolution Generative Adversarial Networks, referred to as SRGAN), an enhanced super-resolution generative adversarial network (Enhanced Super-Resolution Generative Adversarial Networks, referred to as ESRGAN), etc.

[0080] For example, the number of basic residual blocks of ESRGAN is reduced from 16 to 10, and the number of channels is reduced from 256 to 64 to obtain an initial image super-resolution network. The number of basic residual blocks and channels to be cropped can be set according to the actual live broadcast application scenario, and the present invention is not limited to this.

[0081] Step S220, based on the loss information of the enhanced training image and the corresponding image in the enhanced high-definition image set, iteratively update the parameters of the initial image super-resolution network to obtain an image quality enhancement model.

[0082] In an embodiment of the present invention, loss information is obtained based on the features of the enhanced training image and the features of the corresponding enhanced high-definition image, and the parameters of the initial image super-resolution network are iteratively updated based on the loss information until the loss information meets the convergence condition to obtain an image quality enhancement model.

[0083] It can be seen that the embodiment of the present invention reduces the amount of computational effort for image quality enhancement by cutting the number of basic residual blocks and the number of channels, so that the trained image quality enhancement model can meet the real-time requirements of live broadcasting. At the same time, the image quality enhancement model is trained using the enhanced high-definition image set obtained by iteratively enhancing the image quality, effectively improving the image quality enhancement effect of the image quality enhancement model.

[0084] Optionally, before training the image quality enhancement model, it is necessary to generate an enhanced version of the high-definition image set. The following is an explanation of the generation process of the enhanced version of the high-definition image set. Figure 4 , Figure 4 Another schematic diagram of a process of enhancing the video quality provided by an embodiment of the present invention is shown. The enhanced high-definition image set is obtained by:

[0085] Step S300, obtaining a plurality of high-definition videos and an image super-resolution network to be trained; the image super-resolution network to be trained includes a global residual connection.

[0086] In the embodiment of the present invention, recording of high-definition video is an important step in improving the quality of live broadcast. Through scientific recording methods and effective data processing, high-quality high-definition images can be obtained, laying a solid foundation for image quality enhancement and super-resolution processing.

[0087] As a possible implementation method, game live broadcast is used as an example. For games on fixed terminals, the games running at the highest picture quality are recorded to obtain fixed terminal game videos; for games on mobile terminals, the games on mobile terminals are run at the highest picture quality on fixed terminals using a game simulator, and the running games are recorded to obtain mobile terminal game videos; high-definition videos are obtained based on all fixed terminal game videos and all mobile terminal game videos.

[0088] In game live broadcasts, the picture quality directly affects the audience's viewing experience, so improving the live broadcast quality is crucial. To achieve this goal, you first need to obtain game data as high as possible, which provides the basis for image quality enhancement and super-resolution processing. High-definition game data recording is the first step in the entire process, which can provide high-quality training materials for image quality enhancement and super-resolution models. Taking the recording of 4K resolution high-definition video as an example, the following are the specific steps and methods.

[0089] 1) Fixed terminal game recording, such as PC games, can be recorded during normal game running. After opening the game, make sure the game is set to the highest quality and use high-performance hardware devices to support 4K resolution output. In this way, the most delicate picture details in the game can be captured and a fixed terminal game video can be obtained.

[0090] 2) Mobile terminal game recording, such as mobile phone games, due to the difficulty of directly recording 4K resolution, you can use the official game simulator to run mobile games on a fixed terminal (such as a PC). The game simulator can provide higher resolution output and can be recorded more conveniently. Play the game normally in the game simulator, covering different maps and game character scenes as much as possible to ensure that the recorded data is diverse and representative enough to obtain mobile terminal game videos.

[0091] 3) Recording with a high-resolution monitor: Whether it is a fixed terminal game or a mobile terminal game, you should use a high-resolution (such as 4K) monitor for recording. 4K resolution can capture every detail in the game screen and provide the clearest image data. During the recording process, ensure that the game screen is stable and avoid screen jitter and freeze to obtain high-quality recording results.

[0092] Step S310, obtaining a plurality of high-definition video frames, a low-definition image set and a high-definition image set according to each high-definition video; the low-definition image set includes low-definition images with different bit rates.

[0093] In an embodiment of the present invention, the image quality of the image in the high-definition image set is higher than that of the low-definition image in the low-definition image set. The high-definition image may be an image with a high resolution such as 4K resolution or 8K resolution, and the low-definition image may be an image with a resolution lower than 480p. Low-definition images with different bit rates can be encoded according to the high-definition video, and each bit rate will introduce different degrees of compression distortion, thereby simulating the actual image quality when the audience watches the live broadcast under different network conditions. Common bit rates include 6M, 4M, 2M, 500K, etc.

[0094] Step S320: The image super-resolution network to be trained is used as the target network, and the high-definition image set is used as the target image set.

[0095] In the embodiment of the present invention, it is assumed that the open source image super-resolution network selects ESRGAN, which has efficient image enhancement and restoration capabilities, can generate realistic image detail textures, and has good generalization capabilities. In order to better adapt to the image quality enhancement requirements, the following improvements are made to ESRGAN to obtain the image super-resolution network to be trained:

[0096] 1) Remove the last upsampling layer of ESRGAN to ensure that the input and output resolutions are the same.

[0097] 2) Add global residual connections. Add global residual connections from input to output, so that ESRGAN can learn to output the residual of image quality enhancement. The residual of image quality enhancement determines the area of ​​image quality enhancement.

[0098] Step S330, obtaining a training residual according to the randomly acquired low-definition image and the target network.

[0099] Step S340, obtaining a target enhanced image according to the training residual, the preset residual coefficient and the corresponding low-definition image.

[0100] Step S350, iteratively updating the parameters of the target network according to the loss information between the target enhanced image and the corresponding image in the target image set to obtain a trained network.

[0101] In an embodiment of the present invention, in order to improve the generalization ability of the trained network, low-definition images with different bit rates are randomly selected during each training. Such randomness ensures that the trained network will not overfit to the degradation effect of a specific bit rate and can cope with image quality issues at various bit rates.

[0102] Step S360, obtaining a new high-definition image set according to the trained network and each high-definition video frame.

[0103] Step S370, determining whether the image quality enhancement effect of the trained network meets the preset conditions.

[0104] In the embodiment of the present invention, the trained network is used to enhance the image quality of high-definition video frames to obtain a new set of high-definition images. When the image quality enhancement effect gradually becomes weak, it means that the image quality enhancement after enhancement is no longer significant, and it is determined that the image quality enhancement effect of the trained network meets the preset conditions.

[0105] Step S380: When the image quality enhancement effect of the trained network meets the preset conditions, the new high-definition image set is used as the enhanced high-definition image set.

[0106] In the embodiment of the present invention, when the image quality enhancement effect of the trained network meets the preset conditions, an enhanced high-definition image set with the best possible image quality is obtained, and its clarity far exceeds that of the original high-definition video frame.

[0107] Step S390, when the image quality enhancement effect of the trained network does not meet the preset conditions, the trained network is used as the target network, the new high-definition image set is used as the target image set, and the trained network is iteratively trained.

[0108] In an embodiment of the present invention, when the image quality enhancement effect of the trained network does not meet the preset conditions, the trained network is used as the target network, and the new high-definition image set is used as the target image set, and the process returns to step S330 to continue execution, and the trained network is iteratively trained until the image quality enhancement effect of the trained network meets the preset conditions, and the new high-definition image set is used as the enhanced high-definition image set.

[0109] It should be noted that the embodiment of the present invention adopts a gradually adjusted training strategy, starting from a larger learning rate and gradually reducing it to ensure stable convergence of the target network. That is, when the target network is first trained, a larger learning rate is set to facilitate rapid adjustment of the parameters of the target network; as the training continues to iterate, the learning rate is gradually reduced to gradually fine-tune the parameters of the target network.

[0110] It can be seen that the embodiment of the present invention uses low-definition images with different bit rates to iteratively train the image super-resolution network to be trained, which can better simulate the actual situation of live broadcast under different network conditions and improve the generalization ability of the image super-resolution network to be trained. And the image quality enhancement effect is adjusted by using the preset residual coefficient, and the image quality enhancement effect of the trained network is continuously improved, so as to obtain an enhanced version of the high-definition image set with excellent image quality enhancement effect, which can enhance the clarity of the high-definition image and ultimately achieve the purpose of improving the image quality.

[0111] Optionally, in practical applications, low-definition images and high-definition images are generated by introducing different bit rates and sharpening processing, so as to improve the generalization ability and image quality enhancement effect of the trained network. Figure 4 The sub-steps of step S310 may include:

[0112] Each high-definition video is encoded according to different training bit rates to obtain multiple low-definition videos of each high-definition video; each high-definition video and the multiple low-definition videos corresponding to the high-definition video are framed according to a preset frame extraction interval to obtain multiple high-definition video frames corresponding to the high-definition video and multiple low-definition images corresponding to each low-definition video; a low-definition image set is obtained based on all the low-definition images; each high-definition video frame is sharpened to obtain a sharpened image corresponding to each high-definition video frame, and a high-definition image set is obtained based on all the sharpened images.

[0113] In an embodiment of the present invention, after recording a high-definition video, each high-definition video is encoded according to different preset bit rates to obtain corresponding low-definition videos with different bit rates. Aligned video frames are extracted from the high-definition video and multiple low-definition videos with different bit rates corresponding to the high-definition video according to a preset frame extraction interval to obtain multiple high-definition video frames corresponding to the high-definition video and multiple low-definition images corresponding to the low-definition videos with different bit rates, so that each low-definition image has a corresponding high-definition video frame. The aligned low-definition images and high-definition video frames ensure the consistency of the training data, which is convenient for the model to learn the conversion relationship between low-definition images and high-definition video frames.

[0114] It should be noted that the purpose of frame extraction is to extract static high-definition images. Different time points and scenes can be selected when extracting frames to ensure the diversity and comprehensiveness of training data. These images can be used to train image quality enhancement models and obtain enhanced high-definition image sets.

[0115] Although the high-definition video frames obtained after frame extraction have extremely high resolution and clarity, they are not enough to meet the needs of all application scenarios. Many classic images or videos are not as good as the latest images because they were shot earlier. Even if they are recorded by high-resolution (such as 4K) equipment, their image details and clarity will be limited by the original rendering quality of the image.

[0116] Therefore, relying solely on the original recorded high-definition video is not enough to fully reflect the high-quality visual effects. On high-definition display devices, users have higher and higher expectations for image quality. In order to ensure that users get the best visual enjoyment when watching and experiencing, the embodiment of the present invention further enhances the high-definition video to ensure that the enhanced high-definition image can achieve higher visual effects and quality standards.

[0117] By sharpening each high-definition video frame, the clarity and contrast of the high-definition video frame are improved, a corresponding sharpened image is obtained, and a high-definition image set is obtained based on all the sharpened images. Sharpening is a technology that enhances image details. Common sharpening methods include unsharp mask (USM), high pass filter, Laplacian sharpening, etc. Commonly used software tools include Adobe Photoshop, Adobe Lightroom, GIMP, etc. The sharpening method can be selected according to the actual application scenario, and the present invention is not limited to this.

[0118] As a possible implementation method, the process of obtaining a sharpened image is described by taking the USM sharpening technology as an example. USM sharpening blurs the image and then compares the blurred image with the original image to enhance the contrast of the image edge and achieve a sharpening effect. This method can not only enhance the clarity of the image, but also improve the layering and detail performance of the image to a certain extent. Assume that the USM sharpening parameters are set to:

[0119] 1) Amount set to 50%: This parameter controls the strength of the sharpening effect. 50% is a moderate value that can significantly enhance the edges of the image without over-sharpening.

[0120] 2) Set the radius to 4 pixels: The radius parameter determines the range of the sharpening effect. The 4-pixel setting can ensure detail enhancement while avoiding excessive sharpening artifacts in the image.

[0121] 3) Set the threshold to 0: The threshold parameter controls the application range of the sharpening effect. Setting it to 0 means that all pixels are sharpened, which ensures the overall sharpening effect of the image.

[0122] By setting the above parameters, the sharpened image after USM sharpening can significantly improve the clarity and detail of the image, and then a high-definition image set is obtained based on all the sharpened images. The high-definition image set retains the high resolution of the original high-definition video, and further improves the visual effect and detail of the image through sharpening processing.

[0123] It can be seen that the embodiments of the present invention introduce different bit rates through coding degradation, which can quickly simulate the actual situation of the audience watching the live broadcast under different network conditions, ensure that the network can cope with various complex low-quality images, and lay a solid foundation for achieving high-quality live broadcast quality enhancement. By sharpening the high-definition video frames, the enhanced high-definition images perform better in detail restoration, the details are clearer and more realistic, and the generated texture is more natural, avoiding the common problems of over-sharpening and over-smoothing. In different types of live broadcast scenes, the trained network has good generalization ability and the image quality enhancement effect is more stable.

[0124] Optionally, in practical applications, by performing different degradation processing on low-definition images, the training data is diversified, thereby effectively improving the generalization ability of the target network. Figure 4 The sub-steps of step S330 may include:

[0125] Random noise is added to a randomly acquired low-definition image and / or random blurring is performed to obtain a degraded image; the degraded image is input into the target network to obtain a training residual.

[0126] In an embodiment of the present invention, during the training of the target network, low-definition images of different bit rates are randomly acquired, and degradation processing is performed on the randomly acquired low-definition images, such as adding random noise to the randomly acquired low-definition images, performing random blur processing on the randomly acquired low-definition images, adding random noise to the randomly acquired low-definition images and performing random blur processing. After the degradation processing, a sharpened image corresponding to the low-definition image is obtained, and the degraded image is used to train the target image to obtain a training residual.

[0127] It can be seen that the embodiment of the present invention uses degraded images with different noise, blur and bit rate to train the target network, so as to diversify the training data, thereby ensuring that the target network can learn a wider range of image degradation features. The degradation of each training will be different. This dynamic change enables the target network to better perform image quality enhancement and super-resolution processing when facing unknown low-quality images.

[0128] Optionally, in actual applications, the preset residual coefficient is adjusted according to the image quality enhancement effect so as to dynamically adjust the image quality enhancement degree. Figure 4 The sub-steps of step S340 may include:

[0129] The training residual is multiplied by the preset residual coefficient to obtain a residual vector; the residual vector and the feature vector of the low-definition image are summed to obtain a target enhancement vector, and a target enhanced image is generated according to the target enhancement vector.

[0130] In the embodiment of the present invention, the user dynamically adjusts the preset residual coefficient of the next round of training according to the image quality enhancement effect corresponding to the high-definition image set generated after each round of training, and uses the preset residual coefficient to fine-tune the training residual to further adjust the degree of image quality enhancement. For example, when the preset residual coefficient is set to 1, the training residual is fully enhanced by default; when the preset residual coefficient is set to 0.2, it means that the image quality enhancement effect is weakened.

[0131] It can be seen that the embodiment of the present invention dynamically adjusts the preset residual coefficients of each round of training according to the image quality enhancement effect, and adjusts the degree of image quality enhancement based on the training residual and the preset residual coefficient, while ensuring the image quality enhancement effect, effectively avoiding the undesirable phenomenon of multiple enhancement.

[0132] Optionally, in actual applications, the trained network is used to enhance the quality of high-definition image frames, and the high-definition images with enhanced quality are continuously used to train the network, ultimately obtaining an enhanced high-definition image set with higher quality. Figure 4 The sub-steps of step S360 may include:

[0133] Each high-definition video frame is input into the trained network to obtain the target residual corresponding to the high-definition video frame; a high-definition image is obtained according to each target residual, a preset residual coefficient and the corresponding high-definition video frame; and a new high-definition image set is obtained according to all the high-definition images.

[0134] In an embodiment of the present invention, the trained network is used to perform image quality enhancement on each high-definition video frame to obtain a target residual corresponding to each high-definition video frame, and a high-definition image corresponding to the high-definition video frame is obtained based on each target residual, a preset residual coefficient and the corresponding high-definition video frame, and a new high-definition image set is obtained based on all the high-definition images.

[0135] It can be seen that the embodiment of the present invention uses the network after each round of training to generate a high-definition image set for the next round of training, iteratively generates a high-definition image set with better image quality, and iteratively trains the target network. By repeatedly iteratively updating the high-definition image set used for training and the target network to be trained, the image quality of the high-definition image set and the image quality enhancement capability of the target network are continuously improved until an enhanced version of the image set is obtained.

[0136] Based on the same inventive concept, the embodiment of the present invention also provides a video quality enhancement device. Its basic principle and technical effects are the same as those of the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiment.

[0137] Please refer to Figure 5, Figure 5 FIG. 4 is a block diagram of a video quality enhancement device 400 provided in an embodiment of the present invention. The video quality enhancement device 400 includes a decoding module 401 , an enhancement module 402 and an encoding module 403 .

[0138] The decoding module 401 is used to decode the live video to obtain multiple original video frames.

[0139] The enhancement module 402 is used to input the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is trained using a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of the video frames in the high-definition video.

[0140] The encoding module 403 is used to encode the enhanced video frame to obtain the enhanced video.

[0141] In summary, the embodiments of the present invention provide a video quality enhancement device, which performs multiple trainings of superimposed quality enhancement effects on high-definition videos, and finally obtains an enhanced version of a high-definition image set with improved quality, and uses the enhanced version of the high-definition image set to train a quality enhancement model, thereby effectively improving the quality enhancement capability of the quality enhancement model. The quality of the original video frames in the live video is enhanced by the quality enhancement model, and enhanced video frames with significantly improved quality are obtained. Since the quality of the enhanced video frames is significantly improved, the clarity of the live picture can be guaranteed even when the live bit rate is reduced. Since reducing the bit rate can effectively reduce bit rate redundancy and significantly save bandwidth costs, the network bandwidth requirements for viewers watching ultra-high-definition live broadcasts are reduced, allowing viewers to watch ultra-high-definition live broadcasts more smoothly, significantly improving user experience.

[0142] Optionally, the enhancement module 402 is also used to determine a low-definition training image based on a low-definition image set; the low-definition training image is a low-definition image in the low-definition image set, or an image obtained by downsampling a low-definition image in the low-definition image set; the low-definition training image is input into an initial image super-resolution network to obtain a corresponding enhanced training image; the initial image super-resolution network is obtained by cropping the number of basic residual blocks and the number of channels of an open source image super-resolution network; based on the loss information of the enhanced training image and the corresponding image in the enhanced high-definition image set, the parameters of the initial image super-resolution network are iteratively updated to obtain a picture quality enhancement model.

[0143] Optionally, the enhancement module 402 is further used to obtain multiple high-definition videos and image super-resolution networks to be trained; the image super-resolution network to be trained includes a global residual connection; according to each high-definition video, multiple high-definition video frames, low-definition image sets and high-definition image sets are obtained; the low-definition image sets include low-definition images with different bit rates; the image super-resolution network to be trained is used as a target network, and the high-definition image set is used as a target image set;

[0144] According to the randomly acquired low-definition image and the target network, a training residual is obtained; according to the training residual, the preset residual coefficient and the corresponding low-definition image, a target enhanced image is obtained; according to the loss information between the target enhanced image and the corresponding image in the target image set, the parameters of the target network are iteratively updated to obtain a trained network; according to the trained network and each high-definition video frame, a new high-definition image set is obtained;

[0145] When the image quality enhancement effect of the trained network meets the preset conditions, the new high-definition image set is used as the enhanced high-definition image set; when the image quality enhancement effect of the trained network does not meet the preset conditions, the trained network is used as the target network, the new high-definition image set is used as the target image set, and the trained network is iteratively trained.

[0146] Optionally, the enhancement module 402 is specifically used to encode each high-definition video according to different training bit rates to obtain multiple low-definition videos of each high-definition video; extract frames for each high-definition video and multiple low-definition videos corresponding to the high-definition video according to a preset frame extraction interval to obtain multiple high-definition video frames corresponding to the high-definition video and multiple low-definition images corresponding to each low-definition video; obtain a low-definition image set based on all the low-definition images; sharpen each high-definition video frame to obtain a sharpened image corresponding to each high-definition video frame, and obtain a high-definition image set based on all the sharpened images.

[0147] Optionally, the enhancement module 402 is specifically used to add random noise and / or perform random blur processing on the randomly acquired low-definition image to obtain a degraded image; and input the degraded image into the target network to obtain a training residual.

[0148] Optionally, the enhancement module 402 is specifically configured to multiply the training residual by a preset residual coefficient to obtain a residual vector; sum the residual vector and a feature vector of the low-definition image to obtain a target enhancement vector; and generate a target enhanced image according to the target enhancement vector.

[0149] Optionally, the enhancement module 402 is specifically used to input each high-definition video frame into the trained network to obtain a target residual corresponding to the high-definition video frame; obtain a high-definition image based on each target residual, a preset residual coefficient and the corresponding high-definition video frame; and obtain a new high-definition image set based on all high-definition images.

[0150] Please refer to Figure 6 , Figure 6A block diagram of an electronic device 100 provided in an embodiment of the present invention. The electronic device 100 may be any device having an image processing function, such as a personal computer, a laptop computer, a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0151] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0152] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the video quality enhancement method disclosed in the above embodiments can be implemented.

[0153] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0154] It should be understood that Figure 6 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0155] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the video quality enhancement method disclosed in the above embodiments is implemented.

[0156] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0158] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for enhancing video quality, characterized in that: The method comprises: Decode the live video to obtain multiple original video frames; Inputting the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is obtained by training a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of video frames in the high-definition video; The enhanced video frame is encoded to obtain an enhanced video.

2. The video quality enhancement method according to claim 1, characterized in that: The image quality enhancement model is trained in the following way: Determining a low-definition training image according to the low-definition image set; the low-definition training image is a low-definition image in the low-definition image set, or an image obtained by downsampling a low-definition image in the low-definition image set; Inputting the low-resolution training image into an initial image super-resolution network to obtain a corresponding enhanced training image; The initial image super-resolution network is obtained by trimming the number of basic residual blocks and the number of channels of an open source image super-resolution network; Based on the loss information of the enhanced training image and the corresponding image in the enhanced high-definition image set, the parameters of the initial image super-resolution network are iteratively updated to obtain the image quality enhancement model.

3. The video quality enhancement method according to any one of claims 1-2, characterized in that: The enhanced high-definition image set is obtained by: Acquire multiple high-definition videos and a to-be-trained image super-resolution network; the to-be-trained image super-resolution network includes a global residual connection; According to each high-definition video, a plurality of high-definition video frames, the low-definition image set and the high-definition image set are obtained; the low-definition image set includes low-definition images with different bit rates; Using the image super-resolution network to be trained as a target network and the high-definition image set as a target image set; Obtaining a training residual according to the randomly acquired low-definition image and the target network; Obtaining a target enhanced image according to the training residual, the preset residual coefficient and the corresponding low-definition image; Iteratively updating the parameters of the target network according to the loss information between the target enhanced image and the corresponding image in the target image set to obtain a trained network; Obtain a new high-definition image set according to the trained network and each of the high-definition video frames; When the image quality enhancement effect of the trained network meets the preset conditions, the new high-definition image set is used as the enhanced high-definition image set; When the image quality enhancement effect of the trained network does not meet the preset conditions, the trained network is used as the target network, the new high-definition image set is used as the target image set, and the trained network is iteratively trained.

4. The video quality enhancement method according to claim 3, characterized in that: The step of obtaining a plurality of high-definition video frames, the low-definition image set and the high-definition image set according to each high-definition video comprises: Encoding each of the high-definition videos according to different training bit rates to obtain multiple low-definition videos of each of the high-definition videos; Extract frames of each of the high-definition videos and the multiple low-definition videos corresponding to the high-definition videos according to a preset frame extraction interval, and obtain multiple high-definition video frames corresponding to the high-definition videos and multiple low-definition images corresponding to each low-definition video; Obtaining the low-definition image set according to all the low-definition images; Each of the high-definition video frames is sharpened to obtain a sharpened image corresponding to each of the high-definition video frames, and the high-definition image set is obtained based on all of the sharpened images.

5. The video quality enhancement method according to claim 3, characterized in that: The step of obtaining a training residual according to the randomly acquired low-definition image and the target network includes: Adding random noise and / or performing random blur processing on the randomly acquired low-definition image to obtain a degraded image; The degraded image is input into the target network to obtain a training residual.

6. The video quality enhancement method according to claim 3, characterized in that: The step of obtaining a target enhanced image according to the training residual, the preset residual coefficient and the corresponding low-definition image comprises: Multiplying the training residual by the preset residual coefficient to obtain a residual vector; The residual vector and the feature vector of the low-definition image are summed to obtain a target enhancement vector, and the target enhanced image is generated according to the target enhancement vector.

7. The video quality enhancement method according to claim 3, characterized in that: The step of obtaining a new high-definition image set according to the trained network and each of the high-definition video frames includes: Inputting each of the high-definition video frames into the trained network to obtain a target residual corresponding to the high-definition video frame; Obtaining a high-definition image according to each of the target residuals, the preset residual coefficients and the corresponding high-definition video frames; A new high-definition image set is obtained based on all the high-definition images.

8. A video quality enhancement device, characterized in that: The device comprises: A decoding module, used to decode the live video to obtain multiple original video frames; An enhancement module is used to input the original video frame into a pre-trained image quality enhancement model to obtain an enhanced video frame; the image quality enhancement model is trained using a low-definition image set and an enhanced high-definition image set corresponding to the high-definition video; the enhanced high-definition image set is obtained by iteratively enhancing the image quality of video frames in the high-definition video; The encoding module is used to encode the enhanced video frame to obtain an enhanced video.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the video quality enhancement method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video quality enhancement method according to any one of claims 1 to 7 is implemented.

Images