Model training, video quality improvement method, device and computer equipment

Abstract

The application relates to a model training and video quality improvement method and device and a computer device, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a first video sample and a second video sample of a to-be-trained video quality improvement model; training a to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model; updating the to-be-trained degradation model in the to-be-trained video quality improvement model into the trained degradation model, to obtain an updated video quality improvement model; training a to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model; and updating the to-be-trained super-resolution model in the updated video quality improvement model into the trained super-resolution model, to obtain a trained video quality improvement model. The method can effectively improve the video quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a model training method and device, a video quality improvement method and device, and a computer device. BACKGROUND

[0002] With the development of video processing technology, real-world video super-resolution technology has emerged. By studying the degradation model, the real-world video degradation process can be simulated, and the low-resolution video obtained by degradation can be used to construct a training data set to improve the effect of video super-resolution processing.

[0003] In the prior art, a deep learning network is used to simulate the degradation of real high-resolution video data to generate low-resolution video, such as bicubic down-sampling, kernel estimation, and RealBASICVSR (real video super-resolution model) processing methods. Among them, bicubic down-sampling is a relatively rough linear degradation, and the degradation performance is poor; the kernel estimation method can only simulate limited degradation modes, which is usually limited to known degradation modes; the convolutional network usually focuses on local information, and the long-distance feature extraction capability is insufficient, which cannot fully utilize the effective information of the video frame, and the accuracy is insufficient.

[0004] Therefore, the current video quality improvement technology has the problem of limited video quality improvement. SUMMARY

[0005] Therefore, it is necessary to provide a model training method and device, a video quality improvement method and device, a computer device, a computer readable storage medium, and a computer program product capable of improving video quality in view of the above technical problems.

[0006] In a first aspect, the present application provides a model training method for video quality improvement. The method comprises:

[0007] obtaining a first video sample and a second video sample of a to-be-trained video quality improvement model; the to-be-trained video quality improvement model comprises a to-be-trained degradation model and a to-be-trained super-resolution model; the video resolution of the second video sample is higher than that of the first video sample;

[0008] training the to-be-trained degradation model according to the first video sample and the second video sample to obtain a trained degradation model;

[0009] updating the to-be-trained degradation model in the to-be-trained video quality improvement model to the trained degradation model to obtain an updated video quality improvement model;

[0010] According to the first video sample and the second video sample, a to-be-trained super-resolution model in the updated video quality improvement model is trained to obtain a trained super-resolution model.

[0011] The to-be-trained super-resolution model in the updated video quality improvement model is updated to the trained super-resolution model to obtain a trained video quality improvement model.

[0012] In one of the embodiments, the training of the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample to obtain a trained super-resolution model comprises:

[0013] The first video sample is input to the updated video quality improvement model from a first direction to sequentially obtain a first super-resolution video output by the to-be-trained super-resolution model and a first degradation video output by the trained degradation model, and a first loss value is determined according to the first video sample, the first super-resolution video and the first degradation video.

[0014] The second video sample is input to the updated video quality improvement model from a second direction to sequentially obtain a second degradation video output by the trained degradation model and a second super-resolution video output by the to-be-trained super-resolution model, and a second loss value is determined according to the second video sample, the second degradation video and the second super-resolution video.

[0015] According to the first loss value and the second loss value, the updated video quality improvement model is trained to obtain the trained super-resolution model.

[0016] In one of the embodiments, the determination of the first loss value according to the first video sample, the first super-resolution video and the first degradation video comprises:

[0017] A cycle consistency loss value is determined according to the first video sample and the first degradation video.

[0018] A structure perception loss value is determined according to the first video sample and the first super-resolution video.

[0019] A joint adversarial loss value is determined according to the first super-resolution video and the second video sample.

[0020] The first loss value is obtained according to the cycle consistency loss value, the structure perception loss value and the joint adversarial loss value.

[0021] In one of the embodiments, the training of the to-be-trained degradation model according to the first video sample and the second video sample to obtain a trained degradation model comprises:

[0022] inputting the second video sample into the to-be-trained degradation model to obtain a third degraded video corresponding to the second video sample;

[0023] determining a third loss value according to the second video sample, the third degraded video and the first video sample;

[0024] training the to-be-trained degradation model according to the third loss value to obtain the trained degradation model.

[0025] In one of the embodiments, the inputting of the second video sample into the to-be-trained degradation model to obtain a third degraded video corresponding to the second video sample comprises:

[0026] grouping a video frame sequence corresponding to the second video sample to obtain at least one group of video frames;

[0027] inputting each group of the video frames into the to-be-trained degradation model in parallel, performing feature fusion on the video frames by the to-be-trained degradation model to obtain video frame fusion features;

[0028] performing video reconstruction according to the video frame fusion features to obtain the third degraded video.

[0029] In a second aspect, the present application provides a video quality improvement method. The method comprises:

[0030] grouping a video frame sequence corresponding to a first video to obtain at least one group of first video frames;

[0031] inputting each group of the first video frames into a trained video quality improvement model to obtain second video frames output by the trained video quality improvement model; the trained video quality improvement model is obtained by training according to a model training method for video quality improvement;

[0032] combining the second video frames to obtain a second video; the video resolution of the second video is higher than that of the first video;

[0033] performing parameter adjustment on the second video according to preset video display parameters to obtain a target second video.

[0034] In a third aspect, the present application further provides a model training device for video quality improvement. The device comprises:

[0035] a sample obtaining module, configured to obtain a first video sample and a second video sample of a to-be-trained video quality improvement model; the to-be-trained video quality improvement model comprises a to-be-trained degradation model and a to-be-trained super-resolution model; a video resolution of the second video sample is higher than that of the first video sample;

[0036] a first training module, configured to train the to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model;

[0037] a first updating module, configured to update the to-be-trained degradation model in the to-be-trained video quality improvement model into the trained degradation model, to obtain an updated video quality improvement model;

[0038] a second training module, configured to train the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model;

[0039] a second updating module, configured to update the to-be-trained super-resolution model in the updated video quality improvement model into the trained super-resolution model, to obtain a trained video quality improvement model.

[0040] In a fourth aspect, the present application further provides a video quality improvement device. The device comprises:

[0041] a video grouping module, configured to group a video frame sequence corresponding to a first video, to obtain at least one group of first video frames;

[0042] a quality improvement module, configured to input each group of the first video frames into a trained video quality improvement model, to obtain second video frames output by the trained video quality improvement model; the trained video quality improvement model is obtained according to a model training method for video quality improvement;

[0043] a video combination module, configured to combine the second video frames, to obtain a second video; a video resolution of the second video is higher than that of the first video;

[0044] a parameter adjustment module, configured to perform parameter adjustment on the second video according to preset video display parameters, to obtain a target second video.

[0045] In a fifth aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor realizes the following steps when executing the computer program:

[0046] acquire a first video sample and a second video sample for a to-be-trained video quality improvement model; the to-be-trained video quality improvement model comprises a to-be-trained degradation model and a to-be-trained super-resolution model; a video resolution of the second video sample is higher than a video resolution of the first video sample;

[0047] train the to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model;

[0048] update the to-be-trained degradation model in the to-be-trained video quality improvement model to the trained degradation model, to obtain an updated video quality improvement model;

[0049] train the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model;

[0050] update the to-be-trained super-resolution model in the updated video quality improvement model to the trained super-resolution model, to obtain a trained video quality improvement model.

[0051] In a sixth aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0052] group video frames corresponding to a first video to obtain at least one group of first video frames;

[0053] input each group of first video frames into a trained video quality improvement model to obtain second video frames output by the trained video quality improvement model; the trained video quality improvement model is trained according to a model training method for video quality improvement;

[0054] combine the second video frames to obtain a second video; a video resolution of the second video is higher than a video resolution of the first video;

[0055] perform parameter adjustment on the second video according to preset video display parameters to obtain a target second video.

[0056] In a seventh aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0057] acquire a first video sample and a second video sample for a to-be-trained video quality improvement model; the to-be-trained video quality improvement model comprises a to-be-trained degradation model and a to-be-trained super-resolution model; a video resolution of the second video sample is higher than a video resolution of the first video sample;

[0058] train the to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model;

[0059] update the to-be-trained degradation model in the to-be-trained video quality improvement model to the trained degradation model, to obtain an updated video quality improvement model;

[0060] train the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model;

[0061] update the to-be-trained super-resolution model in the updated video quality improvement model to the trained super-resolution model, to obtain a trained video quality improvement model.

[0062] In an eighth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0063] group video frames corresponding to a first video to obtain at least one group of first video frames;

[0064] input each group of first video frames into a trained video quality improvement model to obtain second video frames output by the trained video quality improvement model; the trained video quality improvement model is trained according to a model training method for video quality improvement;

[0065] combine the second video frames to obtain a second video; a video resolution of the second video is higher than a video resolution of the first video;

[0066] adjust the second video according to preset video display parameters to obtain a target second video.

[0067] In a ninth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the following steps:

[0068] acquire a first video sample and a second video sample for a to-be-trained video quality improvement model; the to-be-trained video quality improvement model comprises a to-be-trained degradation model and a to-be-trained super-resolution model; a video resolution of the second video sample is higher than that of the first video sample;

[0069] train the to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model;

[0070] update the to-be-trained degradation model in the to-be-trained video quality improvement model to the trained degradation model, to obtain an updated video quality improvement model;

[0071] train the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model;

[0072] update the to-be-trained super-resolution model in the updated video quality improvement model to the trained super-resolution model, to obtain a trained video quality improvement model.

[0073] In a tenth aspect, the present application further provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the following steps:

[0074] group video frames corresponding to a first video to obtain at least one group of first video frames;

[0075] input each group of first video frames into a trained video quality improvement model to obtain second video frames output by the trained video quality improvement model; the trained video quality improvement model is trained according to a model training method for video quality improvement;

[0076] combine the second video frames to obtain a second video; a video resolution of the second video is higher than that of the first video;

[0077] adjust the second video according to preset video display parameters to obtain a target second video.

[0078] The aforementioned model training, video quality enhancement method, apparatus, computer equipment, storage medium, and computer program product first acquire first and second video samples of the video quality enhancement model to be trained. Based on these samples, a degraded model is trained to obtain a trained degraded model. Then, the degraded model in the video quality enhancement model is updated with the trained degraded model to obtain an updated video quality enhancement model. Next, based on the first and second video samples, the super-resolution model in the updated video quality enhancement model is trained to obtain a trained super-resolution model. Finally, the super-resolution model in the updated video quality enhancement model is updated with the trained super-resolution model to obtain a trained video quality enhancement model. This allows for the combination of the degraded model and the super-resolution model, forming a bidirectional conversion. During model training, the degraded model and the super-resolution model are jointly trained, enabling continuous optimization of the video quality enhancement model and effectively improving video quality. Attached Figure Description

[0079] Figure 1 This is a flowchart illustrating a model training method for video quality improvement in one embodiment.

[0080] Figure 2 This is a structural block diagram of a video quality improvement model in one embodiment;

[0081] Figure 3 Here is a block diagram of the degenerate model generator in one embodiment;

[0082] Figure 4 This is a block diagram of a super-resolution model generator in one embodiment;

[0083] Figure 5 This is a flowchart illustrating a video quality improvement method in one embodiment;

[0084] Figure 6 This is a block diagram of a bidirectional video perception quality enhancement system based on a transformer generator in one embodiment.

[0085] Figure 7 This is a schematic diagram of the process for obtaining a low-resolution model in one embodiment;

[0086] Figure 8 This is a schematic diagram of the bidirectional video perception conversion process in one embodiment;

[0087] Figure 9 This is a schematic diagram of the video perception quality improvement process in one embodiment;

[0088] Figure 10A structural block diagram of a model training device for video quality improvement in an embodiment;

[0089] Figure 11 A structural block diagram of a video quality improvement device in an embodiment;

[0090] Figure 12 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0091] For the purpose, technical solutions and advantages of the present application to be clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0092] The model training method and the video quality improvement method provided by the embodiments of the present application can be applied to a terminal or a server. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0093] In an embodiment, as shown in Figure 1 A model training method for video quality improvement is provided. Taking the case of applying the method to a terminal for example, the method comprises the following steps:

[0094] In step S110, a first video sample and a second video sample of a video quality improvement model to be trained are obtained. The video quality improvement model to be trained comprises a degradation model to be trained and a super-resolution model to be trained. The video resolution of the second video sample is higher than that of the first video sample.

[0095] The first video sample can be a low-resolution video sample. The second video sample can be a high-resolution video sample.

[0096] The degradation model can be a model for converting a high-resolution video into a low-resolution video. The super-resolution model can be a model for converting a low-resolution video into a high-resolution video.

[0097] In a specific implementation, the to-be-trained degradation model and the to-be-trained super-resolution model can be combined to form a to-be-trained video quality improvement model, low-resolution videos and high-resolution videos are collected and input into a terminal, the terminal can take the obtained low-resolution videos as first video samples and take the obtained high-resolution videos as second video samples. The first video samples and the second video samples can not be matched.

[0098] In actual application, a plurality of low-resolution videos and a plurality of high-resolution videos in the real world can be collected and input into a terminal, the terminal can take the obtained plurality of low-resolution videos as a low-resolution video sample set and take the obtained plurality of high-resolution videos as a high-resolution video sample set. The high-resolution video corresponding to a low-resolution video sample can not be in the high-resolution video sample set, and the low-resolution video corresponding to a high-resolution video sample can not be in the low-resolution video sample set.

[0099] Figure 2 A structural block diagram of a video quality improvement model is provided. According to Figure 2 , the video quality improvement model can adopt a cyclic conversion structure composed of a degradation model and a super-resolution model. The degradation model can be composed of a generator based on a Transformer (attention mechanism-based neural network) and a discriminator based on a convolutional neural network, and the super-resolution model can also be composed of a generator based on a Transformer and a discriminator based on a convolutional neural network. The video quality improvement model can be provided with two input ends connected with the degradation model and the super-resolution model respectively. Through one of the input ends, an original low-resolution video frame sequence can be input into the super-resolution model to obtain an intermediate high-resolution video frame sequence output by the super-resolution model, and then the intermediate high-resolution video frame sequence can be input into the degradation model to obtain a target low-resolution video frame sequence output by the degradation model. Through the other input end, an original high-resolution video frame sequence can be input into the degradation model to obtain an intermediate low-resolution video frame sequence output by the degradation model, and then the intermediate low-resolution video frame sequence can be input into the super-resolution model to obtain a target high-resolution video frame sequence output by the super-resolution model.

[0100] Figure 3 A structural block diagram of a degradation model generator is provided. According to Figure 3Before inputting the degradation model, the high-resolution video can be converted into a high-resolution video frame sequence, the high-resolution video frame sequence is grouped to obtain at least one group of high-resolution video frames, each group of high-resolution video frames is taken as an original high-resolution video frame, and the original high-resolution video frame is input to the degradation model. The degradation model can perform feature extraction on the original high-resolution video frame, and sequentially input the extracted features to a parallel Transformer block, a Concatenate layer (connection layer), and a plurality of Transformer blocks, a PixelShuffle (pixel reorganization) down-sampling layer and a PixelShuffle up-sampling layer, to realize feature fusion of spatial information, temporal information and scale information of the original high-resolution video frame. Then, the video frame can be reconstructed according to the fused features to obtain an initial reconstructed video frame. The original high-resolution video frame can also be subjected to forward optical flow estimation, backward optical flow estimation and bicubic down-sampling respectively, and the obtained forward optical flow estimation result, backward optical flow estimation result and bicubic down-sampling result are superimposed on the initial reconstructed video frame to output a group of low-resolution target reconstructed video frames.

[0101] Figure 4 A structural block diagram of a super-resolution model generator is provided. According to Figure 4 Before inputting the super-resolution model, the low-resolution video can be converted into a low-resolution video frame sequence, the low-resolution video frame sequence is grouped to obtain at least one group of low-resolution video frames, each group of low-resolution video frames is taken as an original low-resolution video frame, and the original low-resolution video frame is input to the super-resolution model. The super-resolution model can perform feature extraction on the original low-resolution video frame, and sequentially input the extracted features to a parallel Transformer block, a Concatenate layer (connection layer), and a plurality of Transformer blocks, a PixelShuffle (pixel reorganization) down-sampling layer and a PixelShuffle up-sampling layer, to realize feature fusion of spatial information, temporal information and scale information of the original low-resolution video frame. Then, the video frame can be reconstructed according to the fused features to obtain an initial reconstructed video frame. The original low-resolution video frame can also be subjected to forward optical flow estimation, backward optical flow estimation and bicubic up-sampling respectively, and the obtained forward optical flow estimation result, backward optical flow estimation result and bicubic up-sampling result are superimposed on the initial reconstructed video frame to output a group of high-resolution target reconstructed video frames.

[0102] In step S120, the degradation model to be trained is trained according to the first video sample and the second video sample, and a trained degradation model is obtained.

[0103] In a specific implementation, the second video sample can be used as a training sample, the first video sample can be used as a sample label, the second video sample can be input into the to-be-trained degradation model to obtain a degraded video output by the to-be-trained degradation model. A loss value can be calculated according to the second video sample, the first video sample and the degraded video. The parameters of the to-be-trained degradation model can be adjusted according to the loss value. The training process of step S120 can be repeated until the loss value converges, and the trained degradation model is obtained.

[0104] In step S130, the to-be-trained degradation model in the to-be-trained video quality improvement model is updated to the trained degradation model, and an updated video quality improvement model is obtained.

[0105] In a specific implementation, the trained degradation model obtained in step S120 can be used to replace the to-be-trained degradation model in the to-be-trained video quality improvement model, and the updated video quality improvement model is obtained.

[0106] In step S140, the to-be-trained super-resolution model in the updated video quality improvement model is trained according to the first video sample and the second video sample, and a trained super-resolution model is obtained.

[0107] In a specific implementation, the first video sample can be input into the updated video quality improvement model from a first direction to sequentially obtain an intermediate high-resolution video output by the to-be-trained super-resolution model and a target low-resolution video output by the trained degradation model. A set of loss values can be calculated according to the first video sample, the intermediate high-resolution video and the target low-resolution video. The second video sample can be input into the updated video quality improvement model from a second direction to sequentially obtain an intermediate low-resolution video output by the trained degradation model and a target high-resolution video output by the to-be-trained super-resolution model. Another set of loss values can be calculated according to the second video sample, the intermediate low-resolution video and the target high-resolution video. The parameters of the to-be-trained super-resolution model can be adjusted according to the two sets of loss values. The training process of step S140 can be repeated until the loss value converges, and the trained super-resolution model is obtained.

[0108] In step S150, the to-be-trained super-resolution model in the updated video quality improvement model is updated to the trained super-resolution model, and a trained video quality improvement model is obtained.

[0109] In a specific implementation, the trained super-resolution model obtained in step S140 can be used to replace the to-be-trained super-resolution model in the updated video quality improvement model, and the trained super-resolution model is obtained.

[0110] The model training method can combine the degradation model and the super-resolution model to form bidirectional conversion, jointly train the degradation model and the super-resolution model in the model training process, realize continuous optimization of the video quality improvement model, and effectively improve the video quality.

[0111] In one embodiment, the step S140 can specifically include: inputting the first video sample into the updated video quality improvement model from a first direction to sequentially obtain a first super-resolution video output by the to-be-trained super-resolution model and a first degradation video output by the trained degradation model, and determining a first loss value according to the first video sample, the first super-resolution video and the first degradation video; inputting the second video sample into the updated video quality improvement model from a second direction to sequentially obtain a second degradation video output by the trained degradation model and a second super-resolution video output by the to-be-trained super-resolution model, and determining a second loss value according to the second video sample, the second degradation video and the second super-resolution video; and training the updated video quality improvement model according to the first loss value and the second loss value to obtain the trained super-resolution model.

[0112] The first direction can be an input direction of inputting the super-resolution model first and then inputting the degradation model. The second direction can be an input direction of inputting the degradation model first and then inputting the super-resolution model.

[0113] The first loss value can include at least one of a cycle consistency loss value, a structure perception loss value and a joint adversarial loss value. The second loss value can also include at least one of the cycle consistency loss value, the structure perception loss value and the joint adversarial loss value.

[0114] In a specific implementation, each group of low-resolution video frames of the low-resolution video sample can be input to the updated video quality improvement model from the first direction to obtain, in sequence, the intermediate high-resolution video frame output by the to-be-trained super-resolution model and the target low-resolution video frame output by the trained degradation model, and a group of loss values can be calculated according to the original low-resolution video frame, the intermediate high-resolution video frame and the target low-resolution video frame. Each group of high-resolution video frames of the high-resolution video sample can also be input to the updated video quality improvement model from the second direction to obtain, in sequence, the intermediate low-resolution video frame output by the trained degradation model and the target high-resolution video frame output by the to-be-trained super-resolution model, and another group of loss values can be calculated according to the original high-resolution video frame, the intermediate low-resolution video frame and the target high-resolution video frame. The parameters of the to-be-trained super-resolution model are adjusted according to the two groups of loss values, and the training process is repeated until the loss values converge, and the trained super-resolution model is obtained.

[0115] In this embodiment, by inputting the first video sample to the updated video quality improvement model from the first direction, the first super-resolution video output by the to-be-trained super-resolution model and the first degradation video output by the trained degradation model are obtained in sequence, the first loss value is determined according to the first video sample, the first super-resolution video and the first degradation video, the second video sample is input to the updated video quality improvement model from the second direction, the second degradation video output by the trained degradation model and the second super-resolution video output by the to-be-trained super-resolution model are obtained in sequence, the second loss value is determined according to the second video sample, the second degradation video and the second super-resolution video, and the trained super-resolution model is obtained by training the updated video quality improvement model according to the first loss value and the second loss value; the high-low resolution video frame cycle conversion structure can be used for cycle conversion, the degradation model and the super-resolution model are jointly trained, the network is continuously optimized, and the video processing quality of the trained super-resolution model is improved.

[0116] In one embodiment, the step of determining the first loss value according to the first video sample, the first super-resolution video and the first degradation video can specifically include: determining a cycle consistency loss value according to the first video sample and the first degradation video; determining a structure perception loss value according to the first video sample and the first super-resolution video; determining a joint adversarial loss value according to the first super-resolution video and the second video sample; and obtaining the first loss value according to the cycle consistency loss value, the structure perception loss value and the joint adversarial loss value.

[0117] In the specific implementation, for a set of original low-resolution video frames input from the first direction, a pixel loss between the original low-resolution video frames and the target low-resolution video frames output by the degradation model can be calculated to obtain a cycle consistency loss; bicubic sampling can also be performed on the original low-resolution video frames to obtain bicubic-sampled video frames, the intermediate high-resolution video frames and the bicubic-sampled video frames are input into a VGG (Visual Geometry Group, super-resolution test sequence) network, and a structure perception loss value is calculated; feature extraction and fusion can also be performed on the intermediate high-resolution video frames and the original high-resolution video frames input from the second direction, and the intermediate high-resolution video frames are input into a discriminator to calculate a joint adversarial loss. The cycle consistency loss value, the structure perception loss value and the joint adversarial loss value calculated are taken as the first loss value.

[0118] It should be noted that for a set of original high-resolution video frames input from the second direction, a pixel loss between the original high-resolution video frames and the target high-resolution video frames output by the super-resolution model can be calculated to obtain a cycle consistency loss; bicubic sampling can also be performed on the original high-resolution video frames to obtain bicubic-sampled video frames, the intermediate low-resolution video frames and the bicubic-sampled video frames are input into a VGG (Visual Geometry Group, super-resolution test sequence) network, and a structure perception loss value is calculated; feature extraction and fusion can also be performed on the intermediate low-resolution video frames and the original low-resolution video frames input from the first direction, and the intermediate low-resolution video frames are input into a discriminator to calculate a joint adversarial loss. The cycle consistency loss value, the structure perception loss value and the joint adversarial loss value calculated are taken as the second loss value.

[0119] In the embodiment, the cycle consistency loss value is determined according to the first video sample and the first degraded video, the structure perception loss value is determined according to the first video sample and the first super-resolution video, the joint adversarial loss value is determined according to the first super-resolution video and the second video sample, and the first loss value is obtained according to the cycle consistency loss value, the structure perception loss value and the joint adversarial loss value. The structure perception loss, the adversarial loss and the cycle consistency of the video frames converted in cycles can be trained to improve the perceptual quality of the video frames.

[0120] In one embodiment, the above step S120 can specifically include: inputting the second video sample into the degradation model to be trained to obtain a third degraded video corresponding to the second video sample; determining a third loss value according to the second video sample, the third degraded video and the first video sample; training the degradation model to be trained according to the third loss value to obtain a trained degradation model.

[0121] In a specific implementation, each group of high-resolution video frames of the high-resolution video sample can be input to the to-be-trained degradation model to obtain low-resolution video frames output by the to-be-trained degradation model. According to the high-resolution video frames, the low-resolution video frames output by the to-be-trained degradation model, and the low-resolution video sample, a group of loss values can be calculated. The parameters of the to-be-trained degradation model are adjusted according to the loss values. The training process is repeated until the loss values converge, and the trained degradation model is obtained.

[0122] In actual applications, the loss value can include at least one of a down-sampling pixel loss value, a structure perception loss value, and a joint adversarial loss value. The bicubic down-sampling can be performed on the high-resolution video frames, and the pixel loss between the bicubic down-sampled video frames and the low-resolution video frames output by the to-be-trained degradation model is calculated to obtain the down-sampling pixel loss value. The bicubic down-sampled video frames can be input to the VGG network to calculate the structure perception loss value. The low-resolution video frames output by the to-be-trained degradation model and the low-resolution video frames in the low-resolution video sample can be extracted and fused, and input to the discriminator to calculate the joint adversarial loss value.

[0123] In the embodiment, the second video sample is input to the to-be-trained degradation model to obtain a third degraded video corresponding to the second video sample. According to the second video sample, the third degraded video, and the first video sample, a third loss value is determined. The to-be-trained degradation model is trained according to the third loss value to obtain a trained degradation model. The degradation model can be trained according to the down-sampling pixel loss, the structure perception loss, and the joint adversarial loss to improve the perceptual quality of the video frames of the degradation model.

[0124] In one embodiment, the step of inputting the second video sample to the to-be-trained degradation model to obtain a third degraded video corresponding to the second video sample can specifically include: grouping the video frame sequence corresponding to the second video sample to obtain at least one group of video frames; inputting each group of video frames to the to-be-trained degradation model in parallel, performing feature fusion on the video frames by the to-be-trained degradation model to obtain video frame fusion features; and performing video reconstruction according to the video frame fusion features to obtain the third degraded video.

[0125] In a specific implementation, the high-resolution video sample can be converted into a high-resolution video frame sequence, the high-resolution video frame sequence can be grouped to obtain at least one group of high-resolution video frames, and each group of high-resolution video frames can be input in parallel to the to-be-trained degradation model. The to-be-trained degradation model can perform feature extraction on the high-resolution video frames, and sequentially input the extracted features to the parallel Transformer block, the Concatenate layer, and the multiple Transformer blocks, the PixelShuffle down-sampling layer, and the PixelShuffle up-sampling layer, to realize feature fusion of spatial information, temporal information, and scale information of the high-resolution video frames. Then, the video frames can be reconstructed according to the fused features to obtain initial reconstructed video frames. The original high-resolution video frames can also be subjected to forward optical flow estimation, backward optical flow estimation, and bicubic down-sampling respectively, and the obtained forward optical flow estimation result, backward optical flow estimation result, and bicubic down-sampling result can be superimposed with the initial reconstructed video frames to output a group of low-resolution reconstructed video frames. The above method can be used to sequentially process each group of high-resolution video frames, and the low-resolution reconstructed video frames obtained after processing can be connected to obtain the degraded video.

[0126] In this embodiment, by grouping the video frame sequence corresponding to the second video sample, at least one group of video frames is obtained, each group of video frames is input in parallel to the to-be-trained degradation model, feature fusion of the video frames is performed by the to-be-trained degradation model to obtain video frame fusion features, video reconstruction is performed according to the video frame fusion features to obtain a third degraded video, the Transformer structure with attention mechanism can be used to refine the spatial, temporal, and scale information of the video frames, the global dependency relationship of the video frames is improved, the local high-frequency information of the video frames is improved by using the convolution discriminator, and the bidirectional optical flow between the video frames can also be combined to simultaneously restore a group of image frames. The bidirectional optical flow between the video frames is combined, the information between the frames is fully utilized, and multiple image frames can be efficiently processed, which has good parallelism.

[0127] In one embodiment, as shown in Figure 5 FIG. 1, a video quality improvement method is provided, which is applied to a terminal as an example and includes the following steps:

[0128] In step S210, a video frame sequence corresponding to a first video is grouped to obtain at least one group of first video frames.

[0129] In step S220, each group of first video frames is input to a trained video quality improvement model to obtain second video frames output by the trained video quality improvement model. The trained video quality improvement model is trained according to a model training method for video quality improvement.

[0130] In step S230, the second video frames are combined to obtain a second video, wherein the video resolution of the second video is higher than that of the first video.

[0131] In step S240, the second video is adjusted according to preset video display parameters to obtain a target second video.

[0132] The video display parameters can be, but are not limited to, the size, brightness and color of the video.

[0133] In a specific implementation, for a low-resolution video that needs to be improved in video quality, the low-resolution video can be converted into a low-resolution video frame sequence, the low-resolution video frame sequence is grouped to obtain at least one group of low-resolution video frames, each group of low-resolution video frames is input into the trained video quality improvement model in parallel, high-resolution video frames output by the trained video quality improvement model are obtained, the obtained multiple groups of high-resolution video frames are combined to obtain a high-resolution video, and the high-resolution video is adjusted according to preset video display parameters, so that a high-resolution video with high perceptual quality suitable for display can be obtained.

[0134] In this embodiment, the video frame sequence corresponding to the first video is grouped to obtain at least one group of first video frames, each group of first video frames is input into the trained video quality improvement model, second video frames output by the trained video quality improvement model are obtained, the second video frames are combined to obtain a second video, the second video is adjusted according to preset video display parameters to obtain a target second video, the video quality improvement model obtained by jointly training the degradation model and the super-resolution model can be used to continuously optimize the improvement of video quality, and the video quality is effectively improved.

[0135] In order to facilitate those skilled in the art to further understand the embodiments of the present application, the following will be described in conjunction with a specific example.

[0136] The application provides a bidirectional video perceptual quality improvement method and system based on a transformer generator, which overcomes the shortcomings of the prior art method of simulating degradation by a convolutional network alone to obtain training video data. The transformer network is used as a generator, combined with a convolutional neural network discriminator, and bidirectional optical flow information is used, which combines the advantages of the transformer network in global dependence and the advantages of the convolutional neural network in local detail feature processing, fully utilizes the information between video frames, effectively generates a low-resolution conversion model, and does not directly use the generated low-resolution data, and combines the degradation network and the super-resolution network to form bidirectional conversion and joint training for continuous optimization. In addition, the application also adopts structure perception loss, adversarial loss and cycle consistency loss to simultaneously train the degradation network and the super-resolution network, and uniformly processes all input frames, has good performance, and fully utilizes resources.

[0137] Figure 6 The application provides a bidirectional video perceptual quality improvement method and system based on a transformer generator, which overcomes the shortcomings of the prior art method of simulating degradation by a convolutional network alone to obtain training video data. The transformer network is used as a generator, combined with a convolutional neural network discriminator, and bidirectional optical flow information is used, which combines the advantages of the transformer network in global dependence and the advantages of the convolutional neural network in local detail feature processing, fully utilizes the information between video frames, effectively generates a low-resolution conversion model, and does not directly use the generated low-resolution data, and combines the degradation network and the super-resolution network to form bidirectional conversion and joint training for continuous optimization. In addition, the application also adopts structure perception loss, adversarial loss and cycle consistency loss to simultaneously train the degradation network and the super-resolution network, and uniformly processes all input frames, has good performance, and fully utilizes resources. Figure 6 The application provides a bidirectional video perceptual quality improvement method and system based on a transformer generator, which overcomes the shortcomings of the prior art method of simulating degradation by a convolutional network alone to obtain training video data. The transformer network is used as a generator, combined with a convolutional neural network discriminator, and bidirectional optical flow information is used, which combines the advantages of the transformer network in global dependence and the advantages of the convolutional neural network in local detail feature processing, fully utilizes the information between video frames, effectively generates a low-resolution conversion model, and does not directly use the generated low-resolution data, and combines the degradation network and the super-resolution network to form bidirectional conversion and joint training for continuous optimization. In addition, the application also adopts structure perception loss, adversarial loss and cycle consistency loss to simultaneously train the degradation network and the super-resolution network, and uniformly processes all input frames, has good performance, and fully utilizes resources.

[0138] The low-resolution model acquisition module 300 is responsible for processing mismatched real high / low-resolution video data and receiving a small piece of video data selected by a user as a sample. The video is extracted as a video sequence, and each group of 7 video frames is processed. The transformer generator network is used to fuse the forward and backward bidirectional optical flow information, obtain refined features, and use the PixelShuffle-based downsampling reconstruction layer to obtain low-resolution video frames. The generated low-resolution video frames and the mismatched real low-resolution video frames are respectively concatenated, and the convolutional discriminator is used to distinguish the two groups of concatenated feature vectors, so as to improve the perceptual authenticity of the degraded video frames. The network is trained by means of the bicubic downsampling pixel loss, the structure perception loss and the adversarial loss. Finally, the degraded transformer network generation model and the sample data are output.

[0139] The bidirectional video perceptual conversion module 400 constructs a bidirectional conversion network, and cycles the matched low-resolution video frames and high-resolution video frames for conversion. The degradation generator network is a low-resolution model acquisition module, and the super-resolution generator network has the same structure as the degradation network, wherein the down-sampling layer of the PixelShuffle of the reconstruction layer is changed to an up-sampling layer. This module concatenates the generated high / low-resolution video frames and the matched input high / low-resolution video frames, respectively, uses high / low-resolution convolution discriminators to discriminate the generated / input feature vectors in the high-resolution domain and the low-resolution domain, respectively, and improves the perceptual authenticity of the video frames after cyclic conversion. The network is trained with the help of cycle consistency loss, structural perceptual loss, and adversarial loss. Finally, a well-trained super-resolution transformer network generation model is obtained, and sample data is obtained from the low-resolution model acquisition module.

[0140] The video perceptual quality improvement module 500 integrates the trained network model and is responsible for processing videos that need to be super-resolved. This module displays sample data, each of which is a real low-resolution video segment selected by the user from the training data, representing a trained network model. This module receives user selection on the display interface, low-resolution video and target video parameters, extracts the corresponding super-resolution model, and converts the user-input video that needs to be super-resolved into a picture sequence, 7 pictures per group, combines the selected model, performs super-resolution through the super-resolution network, and finally combines the super-resolution picture sequence with sufficient detail information to form a video, and adjusts the size and other aspects according to the target video parameters. Finally, a high-perceptual-quality clear video is output.

[0141] Figure 7 The process diagram for obtaining a low-resolution model. According to Figure 7 The low-resolution model acquisition module 300 is responsible for processing unmatched real high / low-resolution video data and delivering sample video segments, and the specific steps are as follows:

[0142] Step S301, obtain the unmatched real high / low-resolution video data uploaded by the user, and obtain the sample low-resolution video segment cut by the user.

[0143] Step S302, process the video data, and convert the high / low-resolution video data into video sequences, 7 frames per group, wherein the high-resolution video frame data is used as input data, and the unmatched low-resolution video frame data is used as label data.

[0144] Step S303, using the transformer generator and the convolutional neural network discriminator to perform degradation modeling, and training until convergence. Among them, the training target is the joint adversarial loss of 7 frame image downsampling pixel loss, structure perception loss and mismatched image.

[0145] Among them, the downsampling pixel loss is to perform bicubic downsampling on the high-resolution video frame, and calculate 7 L1 pixel loss with the generated low-resolution video frame; the structure perception loss is to input 7 pairs of generated / bicubic downsampling video frames into the VGG network to calculate 7 VGG losses; the joint adversarial loss is to respectively extract and fuse features from the generated low-resolution video frame group and the corresponding number of mismatched real low-resolution video frames, and input the two groups of features into the discriminator to calculate the adversarial loss.

[0146] Step S304, output the trained degradation transformer network generation model and sample data.

[0147] Figure 8 A flowchart of a bidirectional video perception conversion. According to Figure 8 , the bidirectional video perception conversion module 400 constructs a bidirectional conversion network, which cyclically converts matched low-resolution video frames and high-resolution video frames, and finally outputs a video perception quality improvement model. The specific steps are as follows:

[0148] Step S401, obtain data, including mismatched real high / low resolution video data, a degradation transformer network generation model, and sample video segments.

[0149] Step S402, cyclically convert high and low resolution video frames, and train until convergence. Among them, the training target of each iteration has two groups, each containing 7 frame image cyclic consistency loss, structure perception loss and joint adversarial loss of mismatched images.

[0150] Among them, the cyclic consistency loss is to calculate 7 L1 pixel loss between the video frames generated after two conversions and the original video frames; the structure perception loss is to input the video frames generated after one conversion and the original video frames after bicubic sampling and size adjustment into the VGG network to calculate 7 VGG losses; the joint adversarial loss is to respectively extract and fuse features from the video frame group generated after one conversion and the corresponding number of mismatched real video frame groups, and input the two groups of features into the discriminator to calculate the adversarial loss.

[0151] Step S403, select the trained super-resolution transformer network generation model as the video perception quality improvement model, use the sample data as the label, and output the model and the label.

[0152] Figure 9 A schematic diagram of a video perceptual quality improvement process. According to Figure 9 , the video perceptual quality improvement module 500 is responsible for improving the perceptual quality of the target low-resolution video, and the specific steps are as follows:

[0153] Step S501, obtaining data, including user input low-resolution video, selected video perceptual quality improvement model and target video parameters.

[0154] Step S502, processing video data, converting low-resolution video data into video picture sequences, every 7 frames as a group, which is used as input data.

[0155] Step S503, calling the video perceptual quality improvement model selected by the user, generating the resolution of each group of input data through the super-resolution transformer generator, increasing the picture sequence details and improving the quality.

[0156] Step S504, combining the super-resolution picture sequence into a video, and adjusting the size and other aspects according to the target video parameters, and finally outputting a clear video with high perceptual quality.

[0157] Figure 3 And Figure 4 The network structure of the transformer generator, in which the PixelShuffle layer in the reconstruction layer of the degradation generator is a down-sampling layer, and the corresponding resize layer is a bicubic down-sampling layer, while the PixelShuffle layer of the super-resolution generator is an up-sampling layer, and the corresponding resize layer is a bicubic up-sampling layer. The transformer basic block is a general basic module, and the discriminator is a convolutional neural network discriminator.

[0158] Figure 2 The overall structure idea is that the real low-resolution image sequence (video frame) is first converted through a super-resolution transformer network, and then converted through a degradation transformer network; the real high-resolution image sequence (video frame) is first converted through a transformer network, and then converted through a super-resolution transformer network.

[0159] The above-mentioned bidirectional video perceptual quality improvement method based on transformer generator uses the transformer structure with attention mechanism to refine the spatial, temporal and scale information of the video frame, establishes and perfects the global dependency relationship of the video frame, and uses the convolutional discriminator to improve the local high-frequency information of the video frame.

[0160] Moreover, since the cyclic conversion is performed using the structure of high and low resolution video frame cyclic conversion, the network is continuously optimized by jointly training the degradation and super-resolution networks.

[0161] In addition, the structure perception loss, the adversarial loss and the cycle consistency of the video frame converted by the cycle conversion are trained, and the perceptual quality of the video frame is improved.

[0162] Finally, the bidirectional optical flow between the video frames is combined, and a group of image frames are restored, the bidirectional optical flow between the video frames is combined, the information between the frames is fully utilized, and multiple image frames can be efficiently processed, and good parallelism is achieved.

[0163] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0164] Based on the same inventive concept, the embodiments of the present application also provide a model training and video quality improvement device for implementing the above-mentioned model training and video quality improvement method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more model training and video quality improvement device embodiments provided below can refer to the limitations of the model training and video quality improvement method described above, which will not be repeated here.

[0165] In one embodiment, as shown in FIG. 6, a model training device for video quality improvement is provided, which includes a sample acquisition module 610, a first training module 620, a first update module 630, a second training module 640 and a second update module 650, wherein: Figure 10 The sample acquisition module 610 is configured to acquire a first video sample and a second video sample of a video quality improvement model to be trained; the video quality improvement model to be trained includes a degradation model to be trained and a super-resolution model to be trained; the video resolution of the second video sample is higher than that of the first video sample;

[0166]

[0167] ​The first training module 620 is configured to train the to-be-trained degradation model according to the first video sample and the second video sample, to obtain a trained degradation model.

[0168] The first updating module 630 is configured to update the to-be-trained degradation model in the to-be-trained video quality improvement model into the trained degradation model, to obtain an updated video quality improvement model.

[0169] The second training module 640 is configured to train the to-be-trained super-resolution model in the updated video quality improvement model according to the first video sample and the second video sample, to obtain a trained super-resolution model.

[0170] The second updating module 650 is configured to update the to-be-trained super-resolution model in the updated video quality improvement model into the trained super-resolution model, to obtain a trained video quality improvement model.

[0171] In an embodiment, the second training module 640 further includes:

[0172] The first loss value module is configured to input the first video sample from a first direction to the updated video quality improvement model, to sequentially obtain a first super-resolution video output by the to-be-trained super-resolution model and a first degradation video output by the trained degradation model, and to determine a first loss value according to the first video sample, the first super-resolution video and the first degradation video.

[0173] The second loss value module is configured to input the second video sample from a second direction to the updated video quality improvement model, to sequentially obtain a second degradation video output by the trained degradation model and a second super-resolution video output by the to-be-trained super-resolution model, and to determine a second loss value according to the second video sample, the second degradation video and the second super-resolution video.

[0174] The video quality improvement model training process is configured to train the updated video quality improvement model according to the first loss value and the second loss value, to obtain the trained super-resolution model.

[0175] In an embodiment, the first loss value module is further configured to determine a cycle consistency loss value according to the first video sample and the first degradation video, to determine a structure perception loss value according to the first video sample and the first super-resolution video, to determine a joint adversarial loss value according to the first super-resolution video and the second video sample, and to obtain the first loss value according to the cycle consistency loss value, the structure perception loss value and the joint adversarial loss value.

[0176] In one embodiment, the first training module 620 further includes:

[0177] The model degradation module is used to input the second video sample into the degradation model to be trained, and obtain the third degradation video corresponding to the second video sample;

[0178] The third loss value module is used to determine a third loss value based on the second video sample, the third degraded video, and the first video sample.

[0179] The model training module is used to train the degenerate model to be trained based on the third loss value, so as to obtain the trained degenerate model.

[0180] In one embodiment, the above-mentioned model degradation module is further configured to group the video frame sequence corresponding to the second video sample to obtain at least one group of video frames; input each group of video frames in parallel to the degradation model to be trained, perform feature fusion on the video frames through the degradation model to be trained to obtain video frame fusion features; and perform video reconstruction based on the video frame fusion features to obtain the third degradation video.

[0181] In one embodiment, such as Figure 11 As shown, a video quality enhancement device is provided, including: a video grouping module 710, a quality enhancement module 720, a video combination module 730, and a parameter adjustment module 740, wherein:

[0182] The video grouping module 710 is used to group the video frame sequence corresponding to the first video to obtain at least one group of the first video frames;

[0183] The quality enhancement module 720 is used to input each group of the first video frames into the trained video quality enhancement model to obtain the second video frames output by the trained video quality enhancement model; the trained video quality enhancement model is trained according to the model training method of any one of claims 1 to 5.

[0184] The video combining module 730 is used to combine the second video frames to obtain a second video; the video resolution of the second video is higher than that of the first video.

[0185] The parameter adjustment module 740 is used to adjust the parameters of the second video according to the preset video display parameters to obtain the target second video.

[0186] The various modules in the model training and video quality improvement apparatus can be implemented by software, hardware, or a combination thereof. The various modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be invoked and executed by a processor to perform operations corresponding to the various modules.

[0187] In an embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram thereof can be as shown in FIG. 1. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, mobile cellular network, NFC (near field communication), or other technologies. The computer program is executed by the processor to implement a model training and video quality improvement method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball, or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc. Figure 12

[0188] Those skilled in the art can understand that the structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Figure 12

[0189] In an embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the steps in the above method embodiments.

[0190] In an embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0191] In an embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0192] ​​It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0193] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0194] The technical features of the above embodiments can be combined in any way. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present disclosure.

[0195] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A model training method for video quality improvement, characterized in that, The method includes: Obtain a first video sample and a second video sample for the video quality improvement model to be trained; the video quality improvement model to be trained includes a degradation model to be trained and a super-resolution model to be trained; the video resolution of the second video sample is higher than that of the first video sample. The degradation model to be trained is trained based on the first video sample and the second video sample to obtain the trained degradation model. The degraded model to be trained in the video quality improvement model to be trained is updated with the trained degraded model to obtain the updated video quality improvement model. Based on the first video sample and the second video sample, the super-resolution model to be trained in the updated video quality improvement model is trained to obtain the trained super-resolution model. The super-resolution model to be trained in the updated video quality improvement model is updated to the trained super-resolution model to obtain the trained video quality improvement model. The step of training the degradation model to be trained based on the first video sample and the second video sample to obtain the trained degradation model includes: The video frame sequence corresponding to the second video sample is grouped to obtain at least one group of video frames; Each group of video frames is input in parallel into the degradation model to be trained, and the video frames are fused using the degradation model to obtain video frame fusion features. Video reconstruction is performed based on the video frame fusion features to obtain initial reconstructed video frames; Forward optical flow estimation, backward optical flow estimation, and bicubic downsampling are performed on the original high-resolution video frames corresponding to the second video sample. The obtained forward optical flow estimation results, backward optical flow estimation results, and bicubic downsampling results are superimposed with the initial reconstructed video frames to obtain the third degraded video. A third loss value is determined based on the second video sample, the third degraded video, and the first video sample; The degenerate model to be trained is trained based on the third loss value to obtain the trained degenerate model.

2. The method according to claim 1, characterized in that, The step of training the super-resolution model to be trained in the updated video quality improvement model based on the first video sample and the second video sample to obtain the trained super-resolution model includes: The first video sample is input from the first direction into the updated video quality improvement model, and the first super-resolution video output by the super-resolution model to be trained and the first degraded video output by the degraded model after training are obtained in sequence. Based on the first video sample, the first super-resolution video and the first degraded video, a first loss value is determined. The second video sample is input from the second direction into the updated video quality improvement model, and the second degraded video output by the trained degradation model and the second super-resolution video output by the super-resolution model to be trained are obtained in sequence. Based on the second video sample, the second degraded video and the second super-resolution video, the second loss value is determined. The updated video quality improvement model is trained based on the first loss value and the second loss value to obtain the trained super-resolution model.

3. The method according to claim 2, characterized in that, The step of determining the first loss value based on the first video sample, the first super-resolution video, and the first degraded video includes: Determine the cycle consistency loss value based on the first video sample and the first degraded video; The structural perception loss value is determined based on the first video sample and the first super-resolution video; Determine the joint adversarial loss value based on the first super-resolution video and the second video sample; The first loss value is obtained based on the cycle consistency loss value, the structure awareness loss value, and the joint adversarial loss value.

4. A method for improving video quality, characterized in that, The method includes: The video frame sequence corresponding to the first video is grouped to obtain at least one group of the first video frames; Each group of the first video frames is input into the trained video quality enhancement model to obtain the second video frame output by the trained video quality enhancement model; the trained video quality enhancement model is trained according to the model training method of any one of claims 1 to 3. The second video frames are combined to obtain a second video; the video resolution of the second video is higher than that of the first video. Based on preset video display parameters, the parameters of the second video are adjusted to obtain the target second video.

5. A model training device for video quality improvement, characterized in that, The device includes: The sample acquisition module is used to acquire first video samples and second video samples of the video quality improvement model to be trained; the video quality improvement model to be trained includes a degradation model to be trained and a super-resolution model to be trained; the video resolution of the second video sample is higher than that of the first video sample. The first training module is used to train the degradation model to be trained based on the first video sample and the second video sample to obtain the trained degradation model. The first update module is used to update the degraded model to be trained in the video quality improvement model to be trained with the trained degraded model, so as to obtain the updated video quality improvement model. The second training module is used to train the super-resolution model to be trained in the updated video quality improvement model based on the first video sample and the second video sample, so as to obtain the trained super-resolution model. The second update module is used to update the super-resolution model to be trained in the updated video quality improvement model to the trained super-resolution model, so as to obtain the trained video quality improvement model. The first training module is specifically used to group the video frame sequence corresponding to the second video sample to obtain at least one group of video frames; input each group of video frames in parallel into the degradation model to be trained, and perform feature fusion on the video frames through the degradation model to be trained to obtain video frame fusion features; perform video reconstruction based on the video frame fusion features to obtain an initial reconstructed video frame; perform forward optical flow estimation, backward optical flow estimation, and bicubic downsampling on the original high-resolution video frame corresponding to the second video sample, and superimpose the obtained forward optical flow estimation results, backward optical flow estimation results, and bicubic downsampling results with the initial reconstructed video frame to obtain a third degradation video; determine a third loss value based on the second video sample, the third degradation video, and the first video sample; and train the degradation model to be trained based on the third loss value to obtain the trained degradation model.

6. A video quality enhancement device, characterized in that, The device includes: The video grouping module is used to group the video frame sequence corresponding to the first video to obtain at least one group of the first video frames; A quality enhancement module is used to input each group of the first video frames into a trained video quality enhancement model to obtain the second video frames output by the trained video quality enhancement model; the trained video quality enhancement model is trained according to the model training method of any one of claims 1 to 3. A video combining module is used to combine the second video frames to obtain a second video; the video resolution of the second video is higher than that of the first video. The parameter adjustment module is used to adjust the parameters of the second video according to the preset video display parameters to obtain the target second video.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 4.

Images