Video image quality enhancement method and device and electronic equipment

Abstract

The invention provides a video image quality enhancement method and device, electronic equipment and a storage medium, and the enhancement method comprises the steps: obtaining a downshifted video stream with a first resolution ratio through sending a video stream issuing request, and obtaining a corresponding original frame set according to the video stream, image quality enhancement is carried out on the original frame set through the image quality enhancement model, the image quality enhancement model carries out resolution compression, image quality enhancement and resolution improvement on the original frame set in sequence, and the target frame set with the second resolution is obtained; and obtaining an enhanced video of a second resolution according to the target frame set. Compared with the prior art, the method has the advantages that resolution compression is further performed in the image quality enhancement model, computing resources required for image quality enhancement can be effectively reduced, and image quality enhancement can be effectively realized under limited computing resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, device and electronic equipment for enhancing video quality. Background Art

[0002] With the continuous development of technology, the popularization of high-resolution devices and 5G network technology, users have an increasing demand for ultra-high-definition and smooth live broadcasts. Users hope that the live broadcast resolution can be as high as possible, with minimal lag and reduced latency. However, the higher the resolution of the live video, the higher the bandwidth required for transmission. The higher the resolution, the more memory is required to enhance the quality of the live video, which in turn increases the burden on the device. Summary of the invention

[0003] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a video quality enhancement method, device and electronic device for effectively achieving image quality enhancement while reducing the computing resource occupation of the device.

[0004] Based on the first aspect of the present application, a video quality enhancement method is provided, the enhancement method comprising:

[0005] Send a video stream download request according to the target resolution;

[0006] Receiving a first-resolution video stream sent in response to the video stream sending request; the first resolution is smaller than the target resolution;

[0007] Acquire an original frame set of the first resolution according to the first resolution video stream;

[0008] The original frame set is input into a pre-trained image quality enhancement model; the image quality enhancement model sequentially processes the original frame set through resolution compression, image quality enhancement, and resolution improvement to obtain a target frame set of a second resolution; the second resolution is greater than or equal to the target resolution;

[0009] An enhanced video with a second resolution is acquired according to the target frame set.

[0010] Optionally, the training of the image quality enhancement model specifically includes:

[0011] Acquire a high-definition training frame set and a low-definition training frame set processed based on the high-definition training frame set;

[0012] Inputting the low-definition training image set into the image quality enhancement model for processing, and obtaining a first enhanced residual frame set and a second enhanced residual frame set respectively;

[0013] The loss of the image quality enhancement model is calculated according to the high-definition training frame set, the low-definition training frame set, the first enhanced residual frame set and the second enhanced residual frame set, and the image quality enhancement model is updated and trained according to the loss.

[0014] Optionally, the image quality enhancement model includes:

[0015] The image quality enhancement network is used to compress the resolution of the input image to obtain a compressed image; extract an enhanced residual of the compressed image, and increase the resolution of the enhanced residual to the same as the resolution of the input image to obtain a first enhanced residual;

[0016] A resolution enhancement network, configured to enhance the resolution of the first enhanced residual to be the same as the resolution of a reference image, thereby obtaining a second enhanced residual, wherein the reference image is obtained by enhancing the resolution of the input image;

[0017] The enhanced output network is used to superimpose the reference image and the second enhanced residual to obtain an enhanced image and output it.

[0018] Optionally, the image quality enhancement network includes a downsampling convolution layer, a feature extraction layer, a feature fusion layer and an upsampling convolution layer;

[0019] The downsampling convolution layer is used to reduce the resolution of the input image and extract primary residual features;

[0020] The feature extraction layer is used to extract the primary residual features to obtain secondary residual features;

[0021] The feature fusion layer is used to perform feature fusion on the primary residual feature and the secondary residual feature to obtain a residual fusion feature;

[0022] The upsampling convolution layer is used to restore the resolution of the residual fusion feature to be consistent with the resolution of the input image, and obtain a residual image as a first enhanced residual according to the restored residual fusion feature and output it.

[0023] Optionally, the low-definition training frame set includes a first low-definition training frame set and a second low-definition training frame set;

[0024] The obtaining of a high-definition training frame set and a low-definition training frame set obtained based on the high-definition training frame set specifically includes:

[0025] Obtaining a high-definition training video stream at the second resolution;

[0026] Reducing the resolution of the high-definition training video stream to obtain a first low-definition training video stream with a first resolution;

[0027] Keeping the resolution of the first low-definition training video stream unchanged, reducing the image quality of the first low-definition training video stream, and obtaining a second low-definition training video stream;

[0028] According to several time points, several video frames are extracted from the high-definition training video stream, the first low-definition training video stream and the second low-definition training video stream, and corresponding high-definition training frame sets, first low-definition training frame sets and second low-definition training frame sets are constructed.

[0029] Optionally, the step of inputting the low-definition training image set into the image quality enhancement model for processing to obtain a first enhanced residual frame set and a second enhanced residual frame set, respectively, specifically includes:

[0030] Inputting the second low-definition training frame set into the image quality enhancement model, and obtaining the corresponding first enhanced residual frame set through the image quality enhancement network in the image quality enhancement model;

[0031] The first enhanced residual frame set is input into the resolution enhancement network in the image quality enhancement model to obtain the corresponding second enhanced residual frame set.

[0032] Optionally, the calculating the loss of the image quality enhancement model according to the high-definition training frame set, the low-definition training frame set, the first enhanced residual frame set, and the second enhanced residual frame set specifically includes:

[0033] Superimposing the image data in the second low-definition training frame set with the image data in the first enhanced residual frame set to obtain a first low-definition enhanced frame set;

[0034] Superimposing the image data in the second low-definition training frame set with the image data in the second enhanced residual frame set to obtain a second low-definition enhanced frame set;

[0035] Calculate a first loss of the first low-definition enhancement frame set and the first low-definition training frame set, and calculate a second loss of the second low-definition enhancement frame set and the high-definition training frame set, and use the first loss and the second loss as the loss of the image quality enhancement model.

[0036] Based on the second aspect of the present application, a video quality enhancement device is provided, the enhancement device comprising:

[0037] A request sending module is used to send a video stream download request according to the target resolution;

[0038] A video receiving module, configured to receive a first resolution video stream sent in response to the video stream sending request; the first resolution is smaller than the target resolution;

[0039] A frame set acquisition module, configured to acquire an original frame set of the first resolution according to the first resolution video stream;

[0040] The image quality enhancement module is used to input the original frame set into a pre-trained image quality enhancement model; the image quality enhancement model sequentially processes the original frame set through resolution compression processing, image quality enhancement processing, and resolution improvement processing to obtain a target frame set with a second resolution; the second resolution is greater than or equal to the target resolution;

[0041] The video restoration module is used to obtain an enhanced video with a second resolution according to the target frame set.

[0042] Based on the third aspect of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video quality enhancement method described in the first aspect above.

[0043] Based on the fourth aspect of the present application, a computer storage medium is provided, on which a computer-readable program is stored. When the computer-readable program is executed, a video quality enhancement method described in the first aspect is implemented.

[0044] According to any one of the aspects provided above, the present application provides a video quality enhancement method, device, and electronic device, which obtains a video stream of a first resolution after downshifting by sending a video stream downlink request, and after obtaining the corresponding original frame set according to the video stream, the original frame set is enhanced by the image quality enhancement model, wherein the image quality enhancement model further compresses the resolution of the original frame set. By performing resolution compression in the image quality enhancement model, the computing resources required for image quality enhancement can be effectively reduced, and thus image quality enhancement can be effectively achieved under limited computing resources; at the same time, after image quality enhancement, the resolution of the target frame set is increased to the second resolution through a resolution enhancement operation, and the second resolution is greater than the target resolution, which can better improve the audience's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an application scenario of the enhancement method provided in this embodiment.

[0046] Figure 2 This is a flowchart of the steps of the enhancement method provided in this embodiment.

[0047] Figure 3 This is a network structure diagram of the image quality enhancement model provided in this embodiment.

[0048] Figure 4 A flowchart of the steps for training the image quality enhancement model provided in this embodiment.

[0049] Figure 5This is a structural diagram of the enhancement device provided in this embodiment.

[0050] Figure 6 This is a device structure diagram of the electronic device provided in this embodiment.

[0051] Figures are labeled: server 100, terminal 200, request sending module 11, video receiving module 12, frame set acquisition module 13, image quality enhancement module 14, video restoration module 15, model training module 16, memory 21, processor 22, bus 23, communication interface 24, image quality enhancement network 31, downsampling convolution layer 311, feature extraction layer 312, feature fusion layer 313, upsampling convolution layer 314, resolution improvement network 32, enhanced output network 33. DETAILED DESCRIPTION

[0052] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Nowadays, with the continuous development of technology, the popularization of high-resolution equipment and 5G network technology, users have an increasing demand for ultra-high-definition smooth live broadcasts and videos. Users hope that the resolution of live broadcasts and videos can be as high as possible, and try not to freeze, while reducing delays. The higher the resolution of the live video stream, the higher the bandwidth required to transmit the live video stream; in order to reduce the bandwidth during image quality transmission, before transmitting the live video stream, the live video stream will be downgraded to reduce the resolution of the live video stream, thereby reducing the bandwidth for transmitting the live video stream. When the playback end receives the live video stream, it will restore the resolution of the live video stream, but the downgrade of the live video stream will reduce the image quality of the video obtained after decoding the live video stream. In order to improve the audience's viewing experience, after the client receives the downgraded live video stream, it is necessary to enhance the image quality of the live video stream, but as the resolution of the live video stream increases, the larger the memory required for the image quality enhancement, the more computing resources are required. For devices with limited computing power, such as mobile devices such as mobile phones, when processing high-resolution image quality enhancement, it is necessary to improve the scale of the image quality super-resolution model. Specific improvements may include cropping, optimization, and INT8 quantization.

[0056] Among them, INT8 quantization is a common technology used to reduce the computational complexity and storage requirements of deep learning models during inference. This technology significantly reduces the size and computation of the model by converting model parameters and activation values ​​from 32-bit floating points (FLOAT32) to 8-bit integers (INT8). Modern processors, especially NPUs and DSPs in mobile devices, are usually optimized for integer operations and can perform 8-bit integer calculations faster. Therefore, INT8 quantization can improve the speed of model inference, and INT8 data occupies four times less memory than FLOAT32, making the model more suitable for running on mobile devices with limited memory, which can significantly reduce memory usage, thereby reducing power consumption and heat generation problems. INT8 quantization training simulates the impact of quantization during the training process.

[0057] INT8 quantization training mainly includes the following two processes:

[0058] 1) Analog quantization in forward pass: The operation of analog quantization is used in the forward pass.

[0059] 2) Back-propagation weight update: The model weights are updated through normal back-propagation so that the model can still learn effective features under quantization conditions.

[0060] However, for small models of image quality super-resolution applied on low computing power, their structures are simple, and the number of parameters and calculations are small, which limits the model's ability to improve image quality. When further INT8 quantization is performed, the model may face a further reduction in image quality improvement capabilities and may introduce some image quality issues. Therefore, it is very important to maintain the image quality capabilities of the cropped model during the INT8 quantization process. INT8 quantization reduces the amount of calculation and memory usage by converting floating-point numbers to 8-bit integers, but this conversion may result in a loss of precision. For image quality super-resolution models, this loss may directly affect the details and clarity of the output image, resulting in a decrease in image quality.

[0061] Therefore, in order to solve the above-mentioned problems of the prior art, this embodiment provides a video quality enhancement method and proposes a quality enhancement model structure, which can effectively implement INT8 quantization training, and can be applied to mobile devices with low computing power, and effectively achieve image quality enhancement under low computing power. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0062] For example, a schematic diagram of an application scenario of a video quality enhancement method provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal 200 has a streaming media playback function and can also have an image processing function.

[0063] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0064] In one practicable manner, the server 100 and the terminal 200 may respectively execute a video quality enhancement method provided in an embodiment of the present application, or, optionally, the video quality enhancement method provided in an embodiment of the present application is partially executed in the server 100 and partially executed in the terminal 200.

[0065] like Figure 2 As shown, this embodiment provides a method for enhancing video quality, and the enhancement method may include:

[0066] S1: Send a video stream download request according to the target resolution;

[0067] In this embodiment, the playback end sends a video stream sending request to the video stream sending end, and the video sending request includes the target resolution of the requested video stream. After receiving the video stream sending request, the video stream sending end will send the video stream to the playback end. However, in order to reduce the bandwidth required for transmitting the video stream, the video stream sending end will downgrade the video stream according to the target resolution, reduce the resolution of the video stream, and send the downgraded video to the playback end. Specifically, the playback end may include playback devices such as TVs, desktop computers, laptops, and mobile devices, and the video stream sending end may include video providers such as video live broadcast operators. In addition, the higher the downshift ratio, the greater the resolution change of the video stream, which will increase the learning of the image quality enhancement capability of the subsequent model, so the downshift ratio will not exceed one-half, that is, the resolution of the downgraded video stream will not be lower than one-half of the original resolution of the video stream.

[0068] It can be understood that in the specific implementation of this step, the user selects the video stream resolution he wants to watch at the playback end. After selecting the resolution, the playback end uses the resolution selected by the user as the target resolution, and sends the target resolution to the video stream sending end through a video stream sending request. After receiving the video stream sending request, the video stream sending end prepares the corresponding video stream according to the target resolution in the video stream sending request, and downshifts the video stream with the target resolution to reduce the resolution of the video stream. For example, if the user needs to watch a video with a resolution of 1080p, the target resolution is 1080p, and the video sending end downshifts the resolution of the video stream to a 720p video stream. The specific resolution after downshifting can be set according to the computing power and network conditions of the playback end.

[0069] S2: receiving a first-resolution video stream sent in response to the video stream sending request;

[0070] The first resolution is smaller than the target resolution.

[0071] As described above, after the video stream sending request sent in step S1 is received by the video sending end, the video stream that is downgraded according to the target resolution will be sent to the playback end. In this embodiment, the video sending end reduces the resolution of the video stream from the target resolution to the first resolution. By downgrading the resolution of the video stream, the bandwidth required for transmitting the video stream to the playback end can be effectively reduced, thereby improving the transmission efficiency and facilitating the smoothness of video playback.

[0072] S3: Acquire an original frame set of the first resolution according to the first resolution video stream;

[0073] Specifically, the image quality super-resolution model can usually only enhance the image quality of a single image at a time, so each frame in the video stream is extracted, and all the extracted video frames are constructed into the original frame set in time sequence. Since each video frame in the original frame set is extracted from the video stream of the first resolution, the resolution of each video frame is the first resolution.

[0074] S4: inputting the original frame set into a pre-trained image quality enhancement model;

[0075] In this embodiment, the image quality enhancement model obtains the target frame set of the second resolution by performing resolution compression, image quality enhancement, and resolution improvement in sequence according to the input original frame set; the second resolution is greater than or equal to the target resolution. Figure 3 As shown, the image quality enhancement model may include:

[0076] The image quality enhancement network 31 is used to compress the resolution of the input image to obtain a compressed image; extract an enhanced residual of the compressed image, and increase the resolution of the enhanced residual to the same as the resolution of the input image to obtain a first enhanced residual;

[0077] Specifically, in this embodiment, the image quality enhancement network 31 may include a downsampling convolution layer 311, a feature extraction layer 312, a feature fusion layer 313 and an upsampling convolution layer 314;

[0078] The downsampling convolution layer 311 is used to reduce the resolution of the input image and extract primary residual features;

[0079] The feature extraction layer 312 is used to extract the primary residual features to obtain secondary residual features;

[0080] The feature fusion layer 313 is used to perform feature fusion on the primary residual feature and the secondary residual feature to obtain a residual fusion feature;

[0081] The upsampling convolution layer 314 is used to restore the resolution of the residual fusion feature to be consistent with the resolution of the input image, and obtain a residual image as a first enhanced residual according to the restored residual fusion feature and output it.

[0082] In a preferred implementation of this embodiment, the downsampling convolution layer 311 includes a downsampling convolution unit and a ReLU (Rectified Linear Unit) unit; wherein the number of convolution kernels of the downsampling convolution unit is 3, the convolution step is 2, and the number of output channels is 16; the downsampling convolution layer 311 reduces the resolution of the input image through the downsampling convolution unit, and extracts the primary residual features of the input image, and adds nonlinear processing to the primary residual features through the ReLU unit.

[0083] Preferably, the downsampling convolution layer 311 reduces the resolution of the input image to half of the original resolution. The adjustment multiples through upsampling and downsampling are usually even numbers, i.e., two times, four times, etc. However, an increase in the adjustment multiple will result in a large difference in resolution before and after adjustment, which will make it difficult to reflect the details of the image quality enhancement in the adjusted resolution. At the same time, it will also increase the difficulty of learning the image quality enhancement model. Therefore, it is preferred to reduce the resolution of the input image to half of the original resolution.

[0084] The feature extraction layer 312 includes several feature extraction convolution units arranged in sequence, wherein, except for the last feature extraction convolution unit, a ReLU unit is arranged after each feature extraction convolution unit, the number of convolution kernels of each feature extraction convolution unit is 3, the convolution step is 1, and the number of output channels is 16; the primary residual features extracted by the downsampling convolution layer 311 are sequentially extracted through several feature extraction convolution units in the feature extraction layer 312, and after each feature extraction through a feature extraction convolution unit, nonlinear processing is added to the extracted features through the corresponding ReLU unit to keep the nonlinear representation of the features and pass them to the next feature extraction convolution unit, and finally output the secondary residual features.

[0085] The setting number of the feature extraction convolution units cannot exceed the preset number, and the specific setting number can be adjusted according to the actual computing power and power consumption of the hardware device.

[0086] The feature fusion layer 313 mainly adds the primary residual features extracted by the downsampling convolution layer 311 and the secondary residual features extracted by the feature extraction layer 312 through the Add operation to obtain the residual fusion features. By fusing features at different levels, feature information at different levels can be comprehensively utilized, thereby improving the image quality enhancement effect without significantly increasing the complexity of the model.

[0087] Since the resolution is reduced in the downsampling convolution layer 311, in order to be able to learn the enhancement capability at the original resolution, in this embodiment, it is necessary to restore the resolution through the upsampling convolution layer 314. Specifically, the upsampling convolution layer 314 may include an upsampling convolution unit and a DepthToSpace unit, wherein the number of convolution kernels of the upsampling convolution unit is 3, the convolution step is 2, and the number of output channels is 4 times the number of input image channels. Then, the number of channels is rearranged through the DepthToSpace unit to achieve upsampling and restore the resolution of the residual fusion feature. Exemplarily, if the downsampling convolution layer 311 reduces the resolution of the input image to half of the original resolution, the upsampling convolution layer 314 doubles the resolution of the residual fusion feature.

[0088] A resolution enhancement network 32, configured to enhance the resolution of the first enhanced residual to be the same as the resolution of a reference image, thereby obtaining a second enhanced residual, wherein the reference image is obtained by enhancing the resolution of the input image;

[0089] It is understandable that because the resolution of the input image is obtained through resolution downshifting, even after being restored through the upsampling convolution layer 314 in the image quality enhancement network 31, it still cannot meet the user's target resolution, so it is necessary to further improve the resolution so that the resolution of the final enhanced image can be no lower than the target resolution, so as to facilitate user adjustments and improve the user's viewing experience.

[0090] In this embodiment, the structure of the resolution enhancement network 32 is similar to the structure of the upsampling convolution layer 314 in the image quality enhancement network 31. Preferably, the resolution enhancement network 32 enhances the first enhanced residual output by the image quality enhancement network 31 to twice the original resolution. As mentioned above, if the resolution enhancement factor is too high, it will affect the learning of the image quality enhancement model, so it is optimal to set the enhancement factor to twice.

[0091] Since the resolution enhancement network 32 outputs the second enhanced residual with enhanced image quality after resolution enhancement, and does not directly output the enhanced image, its purpose is to better realize the learning of image quality enhancement information. Therefore, the obtained second enhanced residual requires a reference image. The second enhanced residual is superimposed with the reference image to obtain the final enhanced image, wherein the reference image needs to have the same content as the input image, but the resolution is greater than that of the input image.

[0092] Therefore, in this embodiment, the reference image can be directly obtained by increasing the resolution of the input image. Specifically, the resolution of the input image can be increased through an upsampling network. The resolution of the first enhanced residual obtained through the image quality enhancement network 31 is the same as the resolution of the input image. Therefore, the resolution increase ratio of the upsampling network is the same as the increase ratio of the upsampling convolution layer 314, ensuring that the resolution of the reference image obtained after the input image passes through the upsampling network is the same as the resolution of the second enhanced residual obtained after the first enhanced residual passes through the resolution enhancement network 32. The network structure of the upsampling network can be set to be the same as the structure of the upsampling convolution layer 314.

[0093] The enhanced output network 33 is used to superimpose the reference image and the second enhanced residual to obtain an enhanced image and output it.

[0094] As described above, the resolution of the reference image is the same as the resolution of the second enhanced residual, so the reference image and the second enhanced residual can be directly superimposed to obtain the enhanced image.

[0095] The specific structure of the above-mentioned image quality enhancement model of this embodiment is actually equivalent to obtaining two output results, one is the image quality enhancement information consistent with the input image resolution, namely, the first enhanced residual, and the other is the image quality enhancement information after resolution super-resolution, namely, the second enhanced residual. Through the two output results, the image quality enhancement and super-resolution capabilities of the image quality enhancement model can be effectively improved; at the same time, in this embodiment, the image quality enhancement model is mainly obtained through a limited convolutional layer setting. Under the premise of ensuring the image quality enhancement and super-resolution capabilities of the image quality enhancement model, the structure of the image quality enhancement model is greatly reduced, and image quality enhancement and resolution super-resolution can be achieved using limited computing resources, so that it can be effectively deployed on devices with lower computing power.

[0096] Furthermore, the image quality enhancement model reduces the resolution of the input image by setting a downsampling convolution layer 311, thereby reducing the amount of calculation for subsequent feature extraction, and simultaneously uses two enhanced residuals for learning, thereby ensuring the ability to enhance image quality while reducing the amount of calculation.

[0097] As mentioned above, INT8 quantization training is not effective for training small-scale models. This embodiment introduces two enhanced residuals to effectively combine INT8 quantization training technology to implement the training of the image quality enhancement model. Specifically, the training of the image quality enhancement model is as follows: Figure 4 As shown, it may specifically include:

[0098] A1: Acquire a high-definition training frame set and a low-definition training frame set processed based on the high-definition training frame set;

[0099] The low-definition training frame set includes a first low-definition training frame set and a second low-definition training frame set.

[0100] In this embodiment, obtaining the high-definition training frame set and the corresponding low-definition training frame set may specifically include:

[0101] Obtaining a high-definition training video stream at the second resolution;

[0102] Reducing the resolution of the high-definition training video stream to obtain a first low-definition training video stream with a first resolution;

[0103] Keeping the resolution of the first low-definition training video stream unchanged, reducing the image quality of the first low-definition training video stream, and obtaining a second low-definition training video stream;

[0104] According to several time points, several video frames are extracted from the high-definition training video stream, the first low-definition training video stream and the second low-definition training video stream, and corresponding high-definition training frame sets, first low-definition training frame sets and second low-definition training frame sets are constructed.

[0105] It is understandable that the acquisition of the high-definition training video stream needs to be the same as the acquisition of the video stream in step S2, so as to ensure that the trained image quality enhancement model can be directly applied to the video decoding link of the video stream to achieve the image quality enhancement of the video stream. Specifically, after the video stream is decoded, the video YUV frame is obtained, which is different from the RGB image processed by the conventional image quality super-resolution model. If the conventional video is directly collected and the RGB frame image in the video is extracted for training, it is actually applied to the video decoding link of this embodiment. It is also necessary to perform YUV to RGB conversion. In order to reduce the time-consuming conversion of this step, during the training process of the image quality enhancement model, the high-definition training video stream of the video decoding link is directly obtained. When the video frame is extracted, the video YUV frame can be directly extracted to construct the high-definition training video stream, which is constructed as the high-definition training frame set. The image quality enhancement model is trained by the video YUV frames in the high-definition training frame set and the corresponding low-definition training frame set, so that the image quality enhancement model can directly enhance the image quality based on the video YUV frame, effectively improving the efficiency of video quality enhancement.

[0106] A2: inputting the low-definition training image set into the image quality enhancement model for processing, and obtaining a first enhanced residual frame set and a second enhanced residual frame set respectively;

[0107] Specifically, referring to the specific structure of the above-mentioned image quality enhancement model, in this embodiment, the second low-definition training frame set is input into the image quality enhancement model, and the corresponding first enhanced residual frame set is obtained through the image quality enhancement network 31 in the image quality enhancement model;

[0108] The first training enhanced residual set is input into the resolution enhancement network 32 in the image quality enhancement model to obtain a corresponding second enhanced residual frame set.

[0109] A3: Calculate the loss of the image quality enhancement model according to the high-definition training frame set, the low-definition training frame set, the first enhanced residual frame set and the second enhanced residual frame set, and update the training of the image quality enhancement model according to the loss.

[0110] Specifically, in this embodiment, the image data in the second low-definition training frame set is superimposed with the image data in the first enhanced residual frame set to obtain a first low-definition enhanced frame set;

[0111] Superimposing the image data in the second low-definition training frame set with the image data in the second enhanced residual frame set to obtain a second low-definition enhanced frame set;

[0112] Calculate a first loss of the first low-definition enhancement frame set and the first low-definition training frame set, and calculate a second loss of the second low-definition enhancement frame set and the high-definition training frame set, and use the first loss and the second loss as the loss of the image quality enhancement model.

[0113] Among them, the loss functions of the first loss and the second loss can be set to mean square error, peak signal-to-noise ratio, structural similarity index, etc., which can be set specifically according to actual needs and are not further limited here.

[0114] It can be understood that the training of the above-mentioned image quality enhancement model is based on two main steps of INT8 quantization training technology, so that the image quality enhancement model finally obtained can effectively improve the speed of model reasoning.

[0115] S5: Acquire an enhanced video with a second resolution according to the target frame set.

[0116] Based on the same inventive concept, this embodiment also provides a video quality enhancement device, such as Figure 5 As shown, the enhancement device comprises:

[0117] A request sending module 11 is used to send a video stream download request according to a target resolution;

[0118] In this embodiment, the request sending module 11 can be used to execute Figure 2As shown in step S1, for a detailed description of the request sending module 11, reference may be made to the description of step S1.

[0119] The video receiving module 12 is used to receive a first resolution video stream sent in response to the video stream sending request; the first resolution is smaller than the target resolution;

[0120] In this embodiment, the video receiving module 12 can be used to perform Figure 2 As shown in step S2, for the specific description of the video receiving module 12, reference may be made to the description of step S2.

[0121] A frame set acquisition module 13, configured to acquire an original frame set of the first resolution according to the first resolution video stream;

[0122] In this embodiment, the frame set acquisition module 13 can be used to perform Figure 2 As shown in step S3, for the detailed description of the frame set acquisition module 13, reference may be made to the description of step S3.

[0123] The image quality enhancement module 14 is used to input the original frame set into a pre-trained image quality enhancement model; the image quality enhancement model sequentially processes the original frame set through resolution compression processing, image quality enhancement processing, and resolution improvement processing to obtain a target frame set with a second resolution; the second resolution is greater than or equal to the target resolution;

[0124] In this embodiment, the image quality enhancement module 14 can be used to perform Figure 2 As shown in step S4, for the specific description of the image quality enhancement module 14, reference may be made to the description of step S4.

[0125] The video restoration module 15 is used to obtain an enhanced video with a second resolution according to the target frame set.

[0126] In this embodiment, the video restoration module 15 can be used to perform Figure 2 As shown in step S5, for the specific description of the video restoration module 15, reference may be made to the description of step S5.

[0127] Furthermore, the enhancement device also includes a model training module 16 for training the image quality enhancement model.

[0128] In this embodiment, the model training module 16 can be used to perform Figure 4 For the detailed description of the model training module 16 , please refer to the description of the steps A1-A3.

[0129] This embodiment also provides an electronic device, Figure 6The structural diagram of the electronic device of this embodiment is shown, which includes a memory 21 and a processor 22. The memory 21 stores computer-readable instructions, and the processor 22 executes the computer-readable instructions to implement the video quality enhancement method of this embodiment.

[0130] Preferably, the electronic device further includes a bus 23 and a communication interface 24 , and the processor 22 , the communication interface 24 and the memory 21 are connected via the bus 23 .

[0131] The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 24 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 23 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus 23 may be divided into an address bus, a data bus, a control bus, etc. (not fully drawn in the figure).

[0132] The processor 22 may be an integrated circuit chip with signal processing capability. In the specific implementation process, each step in the above method embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor 22, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the method of the above embodiment in combination with its hardware.

[0133] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor 22, the computer-executable instructions prompt the processor 22 to implement the above-mentioned video quality enhancement method. The specific implementation can be found in the embodiment, which will not be repeated here.

[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0135] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for enhancing video quality, characterized in that: The enhancement method comprises: Send a video stream download request according to the target resolution; Receiving a first-resolution video stream sent in response to the video stream sending request; the first resolution is smaller than the target resolution; Acquire an original frame set of the first resolution according to the first resolution video stream; The original frame set is input into a pre-trained image quality enhancement model; the image quality enhancement model sequentially processes the original frame set through resolution compression, image quality enhancement, and resolution improvement to obtain a target frame set of a second resolution; the second resolution is greater than or equal to the target resolution; An enhanced video with a second resolution is acquired according to the target frame set.

2. A method for enhancing video quality according to claim 1, characterized in that: The training of the image quality enhancement model specifically includes: Acquire a high-definition training frame set and a low-definition training frame set processed based on the high-definition training frame set; Inputting the low-definition training image set into the image quality enhancement model for processing, and obtaining a first enhanced residual frame set and a second enhanced residual frame set respectively; The loss of the image quality enhancement model is calculated according to the high-definition training frame set, the low-definition training frame set, the first enhanced residual frame set and the second enhanced residual frame set, and the image quality enhancement model is updated and trained according to the loss.

3. A method for enhancing video quality according to claim 2, characterized in that: The image quality enhancement model includes: The image quality enhancement network is used to compress the resolution of the input image to obtain a compressed image; extract an enhanced residual of the compressed image, and increase the resolution of the enhanced residual to the same as the resolution of the input image to obtain a first enhanced residual; A resolution enhancement network, configured to enhance the resolution of the first enhanced residual to be the same as the resolution of a reference image, thereby obtaining a second enhanced residual, wherein the reference image is obtained by enhancing the resolution of the input image; The enhanced output network is used to superimpose the reference image and the second enhanced residual to obtain an enhanced image and output it.

4. A method for enhancing video quality according to claim 3, characterized in that: The image quality enhancement network includes a downsampling convolution layer, a feature extraction layer, a feature fusion layer and an upsampling convolution layer; The downsampling convolution layer is used to reduce the resolution of the input image and extract primary residual features; The feature extraction layer is used to extract the primary residual features to obtain secondary residual features; The feature fusion layer is used to perform feature fusion on the primary residual feature and the secondary residual feature to obtain a residual fusion feature; The upsampling convolution layer is used to restore the resolution of the residual fusion feature to be consistent with the resolution of the input image, and obtain a residual image as a first enhanced residual according to the restored residual fusion feature and output it.

5. A method for enhancing video quality according to any one of claims 3 or 4, characterized in that: The low-definition training frame set includes a first low-definition training frame set and a second low-definition training frame set; The obtaining of a high-definition training frame set and a low-definition training frame set obtained based on the high-definition training frame set specifically includes: Obtaining a high-definition training video stream at the second resolution; Reducing the resolution of the high-definition training video stream to obtain a first low-definition training video stream with a first resolution; Keeping the resolution of the first low-definition training video stream unchanged, reducing the image quality of the first low-definition training video stream, and obtaining a second low-definition training video stream; According to several time points, several video frames are extracted from the high-definition training video stream, the first low-definition training video stream and the second low-definition training video stream, and corresponding high-definition training frame sets, first low-definition training frame sets and second low-definition training frame sets are constructed.

6. A method for enhancing video quality according to claim 5, characterized in that: The step of inputting the low-definition training image set into the image quality enhancement model for processing to obtain a first enhanced residual frame set and a second enhanced residual frame set, respectively, specifically includes: Inputting the second low-definition training frame set into the image quality enhancement model, and obtaining the corresponding first enhanced residual frame set through the image quality enhancement network in the image quality enhancement model; The first enhanced residual frame set is input into the resolution enhancement network in the image quality enhancement model to obtain the corresponding second enhanced residual frame set.

7. The video quality enhancement method according to claim 6, characterized in that: The calculating the loss of the image quality enhancement model according to the high-definition training frame set, the low-definition training frame set, the first enhanced residual frame set, and the second enhanced residual frame set specifically includes: Superimposing the image data in the second low-definition training frame set with the image data in the first enhanced residual frame set to obtain a first low-definition enhanced frame set; Superimposing the image data in the second low-definition training frame set with the image data in the second enhanced residual frame set to obtain a second low-definition enhanced frame set; Calculate a first loss of the first low-definition enhancement frame set and the first low-definition training frame set, and calculate a second loss of the second low-definition enhancement frame set and the high-definition training frame set, and use the first loss and the second loss as the loss of the image quality enhancement model.

8. A video quality enhancement device, characterized in that: The enhancement device comprises: A request sending module is used to send a video stream download request according to the target resolution; A video receiving module, configured to receive a video stream with a first resolution sent down; the first resolution is smaller than the target resolution; A frame set acquisition module, configured to acquire an original frame set of the first resolution according to the video stream of the first resolution; The image quality enhancement module is used to input the original frame set of the first resolution into a pre-trained image quality enhancement model; the image quality enhancement model obtains the target frame set of the second resolution through resolution compression, image quality enhancement, and resolution improvement in sequence according to the input original frame set; the second resolution is greater than or equal to the target resolution; The video restoration module is used to obtain an enhanced video with a second resolution according to the target frame set.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a video quality enhancement method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer-readable program, and when the computer-readable program is executed, the video quality enhancement method described in any one of claims 1 to 7 is implemented.

Images