Training method of video enhancement model, video enhancement method and related equipment

Abstract

The invention provides a training method of a video enhancement model, a video enhancement method and related equipment, and relates to the technical field of video processing. The method comprises the following steps: acquiring a training video set; the training video set comprises a plurality of training videos, the training videos comprise a plurality of training video frame pairs, and each training video frame pair comprises a first training video frame and a second training video frame which are adjacent; performing enhancement processing on the first and second training video frames based on a video enhancement model to obtain first and second processed video frames; performing reconstruction processing on the first processed video frame and the second processed video frame based on a conductive encoder to obtain a reconstructed video frame and code rate information; determining loss information according to the reconstructed video frame and the second training video frame; and training a video enhancement model according to the loss information and the code rate information to obtain a trained video enhancement model. According to the method, a derivable encoder is introduced to train a video enhancement model, the video quality and the coded code rate are considered in the training process, and end-to-end optimization is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video enhancement model training method, a video enhancement method, a video enhancement model training device, a video enhancement device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As the number of users in the short video industry continues to grow, more and more viewers are turning into creators. However, due to the limitations of non-professional shooting equipment, the video clarity is low, and the image processing capabilities of the equipment are insufficient, making it difficult to meet the image quality expectations. For this reason, creators often use video editing software for secondary creation to improve video quality. However, existing video enhancement algorithms focus on improving image clarity, while ignoring the encoding difficulty of the enhanced video, resulting in an increase in the bit rate after encoding and an increase in the bandwidth cost required for transmission.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The present disclosure provides a video enhancement model training method, a video enhancement method, a video enhancement model training device, a video enhancement device, an electronic device, a computer-readable storage medium and a computer program product to overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0005] The present disclosure provides a method for training a video enhancement model, the method comprising: obtaining a training video set; the training video set comprises a plurality of training videos, the training videos comprise a plurality of training video frame pairs, the training video frame pairs comprise adjacent first training video frames and second training video frames; performing enhancement processing on the first training video frame based on the video enhancement model to obtain a first processed video frame; performing enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame; performing reconstruction processing on the first processed video frame and the second processed video frame based on a guideable encoder to obtain a reconstructed video frame and bit rate information; determining loss information based on the reconstructed video frame and the second training video frame; training the video enhancement model based on the loss information and the bit rate information to obtain a trained video enhancement model.

[0006] In some embodiments of the present disclosure, the enhancement processing of the first training video frame based on the video enhancement model to obtain a first processed video frame includes: degrading the first training video frame to obtain a first degraded video frame; inputting the first degraded video frame into the video enhancement model for enhancement processing to obtain a first enhanced video frame; and performing video encoding pre-processing on the first enhanced video frame to obtain the first processed video frame.

[0007] In some embodiments of the present disclosure, the enhancing processing of the second training video frame based on the video enhancement model to obtain a second processed video frame includes: degrading the second training video frame to obtain a second degraded video frame; inputting the second degraded video frame into the video enhancement model for enhancing processing to obtain a second enhanced video frame; and performing video encoding pre-processing on the second enhanced video frame to obtain the second processed video frame.

[0008] In some embodiments of the present disclosure, the method further includes: performing pre-encoding processing on the enhanced video frame based on a derivable video pre-encoding processing module.

[0009] In some embodiments of the present disclosure, the method further includes: performing motion estimation on the first training video frame and the second training video frame to obtain a motion vector of the second training video frame relative to the first training video frame.

[0010] In some embodiments of the present disclosure, the reconstruction processing of the first processed video frame and the second processed video frame based on the steerable encoder to obtain the reconstructed video frame and the bit rate information includes: inputting the first processed video frame, the second processed video frame and the motion vector into the steerable encoder for reconstruction processing to obtain the reconstructed video frame and the bit rate information.

[0011] In some embodiments of the present disclosure, the loss information includes one or more of the following options: fidelity loss information, perception loss information, and image quality loss information.

[0012] In some embodiments of the present disclosure, the video enhancement model includes a basic enhancement model and a fine-tuning enhancement model, the basic enhancement model is a pre-trained enhancement model, and the fine-tuning enhancement model is an enhancement model to be trained; wherein, training the video enhancement model according to the loss information and the bit rate information to obtain a trained video enhancement model includes: training the fine-tuning enhancement module according to the loss information and the bit rate information to obtain a trained fine-tuning enhancement model; connecting the basic enhancement model and the trained fine-tuning enhancement model to obtain the trained video enhancement model.

[0013] The present disclosure provides a video enhancement method, which includes: obtaining a video to be processed; the video to be processed includes multiple video frames to be processed; inputting the video frames to be processed into a trained video enhancement model obtained according to the above-mentioned video enhancement model training method to obtain enhanced video frames corresponding to the video frames to be processed.

[0014] The embodiment of the present disclosure provides a training device for a video enhancement model, the device comprising: a training video acquisition module, configured to acquire a training video set; the training video set comprises a plurality of training videos, the training videos comprise a plurality of training video frame pairs, the training video frame pairs comprise adjacent first training video frames and second training video frames; an enhancement processing module, configured to perform enhancement processing on the first training video frame based on the video enhancement model to obtain a first processed video frame; perform enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame; a reconstruction processing module, configured to perform reconstruction processing on the first processed video frame and the second processed video frame based on a steerable encoder to obtain a reconstructed video frame and bit rate information; a loss determination module, configured to determine loss information based on the reconstructed video frame and the second training video frame; a training module, configured to train the video enhancement model based on the loss information and the bit rate information to obtain a trained video enhancement model.

[0015] The present disclosure provides a video enhancement device, which includes: a module for acquiring a video to be processed, configured to acquire a video to be processed; the video to be processed includes multiple video frames to be processed; and a video enhancement module, configured to input the video frames to be processed into a trained video enhancement model obtained by a training method for a video enhancement model according to any one of claims 1 to 8, to obtain an enhanced video frame corresponding to the video frame to be processed.

[0016] An embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the above-mentioned video enhancement model training method, or to implement the above-mentioned video enhancement method.

[0017] An embodiment of the present disclosure provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the above-mentioned video enhancement model training method, or execute the above-mentioned video enhancement method.

[0018] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements the above-mentioned video enhancement model training method or the above-mentioned video enhancement method when executed by a processor.

[0019] The training method of the video enhancement model provided by the embodiment of the present disclosure introduces a guideable encoder to train the video enhancement model. The video quality and the bit rate after encoding are taken into consideration during the training process, and end-to-end optimization can be achieved. This optimization method is more efficient than optimizing the video enhancement model alone, and ensures the maximization of the overall performance. Since the bit rate information after encoding is taken into consideration during the training process, the trained video enhancement model can maintain a low encoding bit rate as much as possible while improving the video quality, which can significantly reduce the required transmission bandwidth, thereby reducing costs. In addition, by using a training video set containing multiple training video frame pairs during the training process, the video enhancement model can learn more diverse video features and processing methods, which helps to improve the generalization ability of the model and enable it to better adapt to video inputs of different styles and qualities.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram showing an exemplary system architecture of a video enhancement model training method or a video enhancement method to which the present disclosure embodiments can be applied;

[0023] Figure 2 A flowchart of a method for training a video enhancement model according to an embodiment of the present disclosure is shown;

[0024] Figure 3 A network structure diagram for training a video enhancement model according to an embodiment of the present disclosure is shown;

[0025] Figure 4 A flowchart of a training video enhancement model according to an embodiment of the present disclosure is shown;

[0026] Figure 5 A flow chart of a video enhancement method according to an embodiment of the present disclosure is shown;

[0027] Figure 6 A block diagram of a video enhancement model training device according to an embodiment of the present disclosure is shown;

[0028] Figure 7 A block diagram of a video enhancement device according to an embodiment of the present disclosure is shown;

[0029] Figure 8 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0031] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0032] The collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of the user personal information involved in this disclosure are in compliance with the relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0033] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0035] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the term "plurality" refers to two or more; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0036] Figure 1 A schematic diagram of an exemplary system architecture of a video enhancement model training method or a video enhancement method to which the present disclosure embodiments can be applied is shown.

[0037] like Figure 1 As shown, the system architecture may include a server 101, a network 102 and a terminal device 103. The network 102 is used to provide a medium for a communication link between the terminal device 103 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0038] In an exemplary embodiment, the terminal device 103 for data transmission with the server 101 may include but is not limited to a smart phone, a tablet computer, a laptop computer, a smart speaker, a digital assistant, an AR (Augmented Reality) device, a VR (Virtual Reality) device, a smart wearable device, etc., or the terminal device 103 may also be a personal computer, such as a laptop computer and a desktop computer, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0039] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server of a network platform, which can be, for example, a trading platform, a live broadcast platform, a social platform, or an audio platform, etc., which is not limited in the embodiments of the present disclosure. Among them, the server can be a single server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0040] In some embodiments of the present disclosure, the process of server 101 for training a video enhancement model may be: obtaining a training video set; the training video set includes multiple training videos, the training videos include multiple training video frame pairs, the training video frame pairs include adjacent first training video frames and second training video frames; based on the video enhancement model, the first training video frame is enhanced to obtain a first processed video frame; based on the video enhancement model, the second training video frame is enhanced to obtain a second processed video frame; based on the guideable encoder, the first processed video frame and the second processed video frame are reconstructed to obtain a reconstructed video frame and bit rate information; based on the reconstructed video frame and the second training video frame, loss information is determined; based on the loss information and the bit rate information, the video enhancement model is trained to obtain a trained video enhancement model.

[0041] In some embodiments of the present disclosure, the process of server 101 for performing video enhancement may be: obtaining a video to be processed, where the video to be processed includes multiple video frames to be processed; inputting the video frames to be processed into a trained video enhancement model obtained according to the training method of the video enhancement model of an embodiment of the present disclosure, and obtaining enhanced video frames corresponding to the video frames to be processed.

[0042] In addition, it should be noted that Figure 1 What is shown is merely an application environment of the training method of the video enhancement model or the video enhancement method provided by the present disclosure. Figure 1 The number of terminal devices 103, networks 101 and servers 101 in the figure is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs.

[0043] Figure 2 A flowchart of a video enhancement model training method according to an embodiment of the present disclosure is shown. Figure 2 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The server 101 in the embodiment is also as follows Figure 1 The terminal device 103 in Figure 1 The server 101 and the terminal device 103 in the embodiment jointly implement the training method of the video enhancement model, but the present disclosure is not limited thereto. Figure 2 The training method of the video enhancement model provided in the embodiment of the present disclosure includes the following steps.

[0044] Step S210, obtaining a training video set; the training video set includes a plurality of training videos, the training videos include a plurality of training video frame pairs, and the training video frame pairs include a first adjacent training video frame and a second adjacent training video frame.

[0045] The training video set is a video set obtained on a media platform, wherein the media platform may be a short video platform, a social media application, a video sharing website, etc. The training video set includes a plurality of training videos, each training video is composed of a plurality of training video frame pairs, and each training video frame pair includes two adjacent video frames, namely a first training video frame and a second training video frame.

[0046] For a training video, it is composed of a series of training video frames (images). These training video frames are not just static images, but also contain rich dynamic information and inter-frame relationships, such as object movement, scene changes, changes in lighting, etc. In the training process of the video enhancement model, adjacent training video frames provide the model with clues about the dynamic characteristics of the video, allowing the model to learn and understand the temporal continuity and spatial correlation between frames. Specifically, by utilizing the information of adjacent frames, the model can learn how to perform targeted enhancement processing on each frame while maintaining the coherence of the video content.

[0047] Step S220, performing enhancement processing on the first training video frame based on the video enhancement model to obtain a first processed video frame; performing enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame.

[0048] The first training video frame is enhanced based on the initialized video enhancement model to improve its visual effects, such as clarity, contrast, color saturation, etc., to obtain a first processed video frame. Similarly, the second training video frame is enhanced based on the initialized video enhancement model to obtain a second processed video frame.

[0049] Step S230: reconstruct the first processed video frame and the second processed video frame based on the guideable encoder to obtain reconstructed video frames and bit rate information.

[0050] Exemplarily, a differentiable encoder has an encoding function and a reconstruction function (decoding function), which means that the functions used in each module for encoding and reconstruction are differentiable or each module is a neural network, so that the model parameters can be adjusted based on the gradient feedback of the loss information.

[0051] Based on the guideable encoder, the enhanced first processed video frame and the second processed video frame are reconstructed, and the reconstruction process generates a reconstructed video frame (that is, a video frame that is encoded and then decoded), and also generates encoded bit rate information.

[0052] Step S240: determining loss information according to the reconstructed video frame and the second training video frame.

[0053] Exemplarily, the reconstructed video frame is a reconstructed video frame corresponding to the second training video frame. The generation of the reconstructed video frame is a process of encoding and then decoding, which is intended to simulate the compression and decompression process of the video frame in an actual transmission or storage environment.

[0054] The loss information is usually a quantitative indicator used to measure the difference between the two. The smaller the difference, the better the enhancement process preserves the original information while improving the video quality. Specifically, by determining the loss information between the reconstructed video frame and the original second training video frame, the quality loss of the enhanced video frame after encoding can be evaluated.

[0055] Step S250: training a video enhancement model according to the loss information and the bit rate information to obtain a trained video enhancement model.

[0056] The video enhancement model is trained using loss information and bitrate information. During the training process, the model parameters are adjusted through the back propagation algorithm to minimize the loss information (i.e., improve the video quality) and control the bitrate information (i.e., maintain a low encoding bitrate). This training method enables the model to find the best balance between improving video quality and maintaining encoding efficiency. After multiple iterations of training, a trained video enhancement model is obtained, which can effectively improve video quality and reduce transmission costs in practical applications.

[0057] The training method of the video enhancement model provided by the embodiment of the present disclosure introduces a guideable encoder to train the video enhancement model. The video quality and the bit rate after encoding are taken into consideration during the training process, and end-to-end optimization can be achieved. This optimization method is more efficient than optimizing the video enhancement model alone, and ensures the maximization of the overall performance. Since the bit rate information after encoding is taken into consideration during the training process, the trained video enhancement model can maintain a low encoding bit rate as much as possible while improving the video quality, which can significantly reduce the required transmission bandwidth, thereby reducing costs. In addition, by using a training video set containing multiple training video frame pairs during the training process, the video enhancement model can learn more diverse video features and processing methods, which helps to improve the generalization ability of the model and enable it to better adapt to video inputs of different styles and qualities.

[0058] Next, the training method of the video enhancement model in the embodiment of the present disclosure will be further described.

[0059] In some embodiments of the present disclosure, the first training video frame is enhanced based on a video enhancement model to obtain a first processed video frame, including: degrading the first training video frame to obtain a first degraded video frame; inputting the first degraded video frame into the video enhancement model for enhancement processing to obtain a first enhanced video frame; performing video encoding pre-processing on the first enhanced video frame to obtain a first processed video frame.

[0060] In some embodiments of the present disclosure, the second training video frame is enhanced based on a video enhancement model to obtain a second processed video frame, including: degrading the second training video frame to obtain a second degraded video frame; inputting the second degraded video frame into the video enhancement model for enhancement processing to obtain a second enhanced video frame; performing video encoding pre-processing on the second enhanced video frame to obtain a second processed video frame.

[0061] Exemplarily, degradation is a process of simulating various quality losses that may be suffered by a video during transmission or storage. Degradation is achieved through a variety of methods, such as adding Gaussian noise, blurring (using Gaussian blur, median blur, etc.), adjusting resolution (downsampling and then upsampling to simulate distortion), introducing compression artifacts (such as JPEG compression noise), etc. Among them, the parameters of the degradation (such as noise level, blur level, resolution change, etc.) can be adjusted according to actual needs to simulate different degradation scenarios.

[0062] The first training video frame is degraded to obtain a first degraded video frame, and the second training video frame is degraded to obtain a second degraded video frame. Through the degraded processing, the training data is closer to the situation that may be encountered in actual applications, so that the video enhancement model obtained by training can better cope with the video degradation problem in actual applications.

[0063] Exemplarily, the video enhancement model is a full-resolution image-to-image model, specifically, a model that can directly process a full-resolution image (or video frame) as input, and then output an enhanced image (or video frame) of the same resolution.

[0064] In one possible implementation, the video enhancement model includes a deep learning model, such as a convolutional neural network (CNN), a generative adversarial network (GAN) or a variant thereof, which is capable of learning a mapping relationship between degraded video frames and high-quality video frames.

[0065] After obtaining the first degraded video frame, the first degraded video frame is enhanced using the video enhancement model to obtain a first enhanced video frame. And after obtaining the second degraded video frame, the second degraded video frame is enhanced using the video enhancement model to obtain a second enhanced video frame.

[0066] Exemplarily, video encoding pre-processing, also known as analog pre-processing, is a series of pre-processing operations performed on video frames before encoding, aiming to optimize encoding efficiency and video quality. Video encoding pre-processing may include smoothing (to reduce high-frequency noise and details), sharpening (to enhance edges and details), color space conversion (such as from RGB to YUV), etc., and appropriate pre-coding processing operations can be selected according to the characteristics and requirements of the guide encoder. For example, if the guide encoder is sensitive to high-frequency noise, smoothing can be performed to reduce noise; if you want to retain more edge details, sharpening can be performed. Through video encoding pre-processing, the characteristics of the video frame can be adjusted to better meet the needs of the encoder, thereby reducing the bit rate after encoding and maintaining a high video quality, and through appropriate video encoding pre-processing operations, the visual effect of the enhanced video frame can be improved to a certain extent, making it more suitable for viewing or subsequent processing.

[0067] After obtaining the first degraded video frame, the first degraded video frame is subjected to video encoding pre-processing to obtain a first processed video frame. And, after obtaining the second degraded video frame, the second degraded video frame is subjected to video encoding pre-processing to obtain a second processed video frame.

[0068] In the disclosed embodiment, by respectively performing degradation processing, enhancement processing and video encoding pre-processing on the first training video frame and the second training video frame, a first processed video frame and a second processed video frame that are both of high quality and suitable for efficient encoding can be obtained, which helps to train a more robust and efficient video enhancement model and provide better video processing effects in practical applications.

[0069] In some embodiments of the present disclosure, the training method of the video enhancement model further includes: performing pre-encoding processing on the enhanced video frame based on a derivable video encoding pre-processing module.

[0070] Exemplarily, based on the guideable video encoding pre-processing module, the first enhanced video frame is subjected to encoding pre-processing to obtain a first processed video frame; based on the guideable video encoding pre-processing module, the second enhanced video frame is subjected to encoding pre-processing to obtain a second processed video frame.

[0071] The configurable video encoding pre-processing module refers to a module that performs a series of pre-processing operations on video frames before encoding, including but not limited to smoothing, sharpening, etc., in order to improve the video quality or make it more suitable for subsequent encoding processes. The configurable video encoding pre-processing module is configurable, that is, its internal processing flow supports the back propagation algorithm.

[0072] In an exemplary embodiment, the guideable video encoding pre-processing module may include the following functions: smoothing processing, such as Gaussian blur and other algorithms, used to reduce image noise and smooth image details; sharpening processing, such as Unsharp Masking (USM), etc., used to enhance the edges and details of the image and improve the clarity of the image; color adjustment, including color balance, white balance adjustment, etc., used to improve the color performance of the image; feature extraction. It should be noted that feature extraction is usually regarded as a part of image recognition or video analysis, but in some advanced video encoding pre-processing, feature-based preprocessing steps may also be included; adaptive adjustment, that is, automatically adjusting preprocessing parameters according to the characteristics of the input video frame (such as brightness, contrast, noise level, etc.); multi-scale processing, that is, performing multi-scale analysis on the video frame so as to apply different preprocessing strategies at different scales.

[0073] Of course, the derivable video encoding pre-processing module may include other functions in addition to smoothing, sharpening, color adjustment, feature extraction, adaptive adjustment, and multi-scale processing functions, which are not limited in the present disclosure. It is worth noting that the functions included in the derivable video encoding pre-processing module are all rewritten in a derivable form so that the gradient can be calculated during back propagation.

[0074] In a possible implementation, various functions included in the guideable video encoding pre-processing module are designed as independent modules to facilitate replacement and combination during the training process.

[0075] Exemplarily, each module used for preprocessing operation in the guideable video encoding preprocessing module is a neural network, and the guideable video encoding preprocessing module is obtained by pre-training. In the process of training the video enhancement model, the pre-trained guideable video encoding preprocessing module is used to perform pre-encoding processing on the enhanced video frame, thereby reducing the time consumption of training the video enhancement model and reducing the training cost.

[0076] In the disclosed embodiment, a derivable video encoding pre-processing module is used to perform video encoding pre-processing on the enhanced processing frame, so as to facilitate gradient feedback during the training process.

[0077] In some embodiments of the present disclosure, the video enhancement model training method further includes: performing motion estimation on the first training video frame and the second training video frame to obtain a motion vector of the second training video frame relative to the first training video frame.

[0078] In a possible implementation, motion estimation is performed according to the following steps to obtain a motion vector:

[0079] (1) Divide the first training video frame and the second training video frame into several small blocks, such as macroblocks of sizes such as 8x8 and 16x16. These blocks are the basic units of motion estimation.

[0080] (2) Determine the search strategy used when searching the second training video frame for the block that best matches each block in the first training video frame. The search strategy includes full search, three step search (TSS), diamond search (DS), etc.

[0081] (3) Selecting a matching criterion to evaluate the similarity between the blocks in the first training video frame and the blocks in the second training video frame. The matching criterion includes minimum mean square error (MSE), minimum absolute error (MAE), normalized cross correlation (NCC), etc.

[0082] (4) For each block in the first training video frame, find the best matching block in the search area of ​​the second training video frame according to the selected search strategy and matching criteria, and then calculate the displacement between the two blocks, that is, obtain the motion vector of the block.

[0083] (5) Post-processing the obtained motion vectors, such as filtering, median filtering, etc., to reduce the influence of noise and outliers. Motion vector field smoothing technology can also be applied to ensure the spatial consistency of motion vectors of adjacent blocks.

[0084] (6) The motion vector of each block is taken as output. These motion vectors are combined to form the motion vector of the entire video frame, that is, the motion vector of the second training video frame relative to the first training video frame.

[0085] In the disclosed embodiment, motion estimation is performed on the first training video frame and the second training video frame to obtain a motion vector of the second training video frame relative to the first training video frame, so that the motion vector can be used during reconstruction processing through a guideable encoder to achieve more efficient video frame reconstruction.

[0086] In some embodiments of the present disclosure, a first processed video frame and a second processed video frame are reconstructed based on a steerable encoder to obtain reconstructed video frames and bit rate information, including: inputting the first processed video frame, the second processed video frame and the motion vector into the steerable encoder for reconstruction to obtain reconstructed video frames and bit rate information.

[0087] Exemplarily, a differentiable encoder has an encoding function and a reconstruction function (decoding function), which means that the functions used in each module for encoding and reconstruction are differentiable or each module is a neural network, so that the model parameters can be adjusted based on the gradient feedback of the loss information.

[0088] In a possible implementation manner, reconstruction processing is performed according to the following steps to obtain reconstructed video frames and bit rate information:

[0089] (1) Data preprocessing is performed on the first processed video frame, the second processed video frame and the motion vector so that data attributes match the input requirements of the encoder.

[0090] (2) The pre-processed first processed video frame, the second processed video frame and the motion vector are input into a derivable encoder to extract features of the first processed video frame and the second processed video frame. These features may include spatial features (such as texture, edge), temporal features (such as motion information), etc.

[0091] (3) Fusing the motion vector with the extracted features to obtain fused features. Fusion can be performed in a variety of ways, such as using the motion vector as an additional input channel, adding a representation of the motion vector to the feature map, or using the motion vector to guide the transformation of the feature map.

[0092] (4) The fused features are processed by the compression coding part of the derivable encoder to obtain compressed coded data, which may include transformation (such as DCT transformation), quantization, entropy coding and other steps, aiming to remove redundant information and reduce the amount of data.

[0093] (5) The compressed coded data is reconstructed into video frames at the output layer of the encoder. This reconstruction process may include steps such as inverse quantization, inverse transformation, and possible residual compensation. During or after the encoding process, the bit rate information is calculated based on the amount of coded data and coding parameters (such as quantization step size).

[0094] (6) Reconstruct the video frame and bit rate information through the output of the steerable encoder.

[0095] Exemplarily, the steerable encoder is a pre-trained encoder. In the process of training the video enhancement model, the pre-trained steerable encoder is used for reconstruction processing, which reduces the time consumption of training the video enhancement model and reduces the training cost.

[0096] In the disclosed embodiment, the motion vector between the first training video frame and the second training video frame is used when performing reconstruction processing based on the guideable encoder, so that the guideable encoder can more effectively analyze the similarities and differences between adjacent video frames, thereby effectively compressing redundant data, reducing the amount of data that needs to be encoded, and improving encoding efficiency. And the guideable encoder performs reconstruction processing based on the first processed video frame, the second processed video frame and the motion vector, and can generate bit rate information based on the complexity and motion characteristics of the video content. In addition, by using motion vectors for motion compensation, the guideable encoder can better handle the moving parts in the video, which helps to reduce blur, ghosting or jitter caused by inaccurate motion estimation, thereby improving the accuracy of the reconstructed video frame.

[0097] Figure 3FIG. 4 shows a network structure diagram for training a video enhancement model according to an embodiment of the present disclosure. Figure 3 As shown, the network includes a video enhancement model 310, a motion estimation module 320, a video encoding pre-processing module 330 and a steerable encoder 340.

[0098] Exemplarily, the video enhancement model 310 is a full-resolution image-to-image model, which refers to a model that can directly process a full-resolution image (or video frame) as input and then output an enhanced image (or video frame) of the same resolution.

[0099] Exemplarily, the motion estimation module 320 is used to analyze motion information between consecutive video frames and generate motion vectors, wherein the motion vectors describe the motion direction and distance of corresponding pixels or pixel blocks in adjacent frames.

[0100] Exemplarily, the video encoding pre-processing module 330 is also called an analog pre-processing module, which is used to perform a series of pre-processing operations on the video frame before encoding, including but not limited to smoothing, sharpening, etc. The video encoding pre-processing module 330 is configurable, that is, the pre-processing operations provided by the module are configurable, or in other words, each module used for the pre-processing operation is a neural network.

[0101] Exemplarily, the differentiable encoder 340 has an encoding function and a reconstruction function (decoding function), which means that the functions used by each module for encoding and reconstruction are differentiable or each module is a neural network, so that the model parameters can be adjusted based on the gradient feedback of the loss information.

[0102] In some embodiments of the present disclosure, the video enhancement model includes a basic enhancement model and a fine-tuning enhancement model, the basic enhancement model is a pre-trained enhancement model, and the fine-tuning enhancement model is an enhancement model to be trained.

[0103] The basic enhancement model is a pre-trained enhancement model, that is, a model that has been fully trained and has video enhancement capabilities. During the training process of the video enhancement model, the weights of the basic enhancement model are set to be non-trainable, that is, the weights of the basic enhancement model are frozen, or in other words, the parameters of the basic enhancement model are unchanged.

[0104] The fine-tuning enhancement model is an enhancement model to be trained, and can also be called an optimized enhancement model or a small model, that is, it can be a model with low complexity, simple structure, and a small number of parameters. In the training process of the video enhancement model, the weights of the fine-tuning enhancement model are trained, or in other words, the parameters of the fine-tuning training model are adjusted.

[0105] Among them, the basic enhancement model and the fine-tuning enhancement model are both full-resolution image-to-image models, that is, they can directly process full-resolution images (or video frames) and use them as input, and then output an enhanced image (or video frame) of the same resolution.

[0106] In the disclosed embodiments, the basic enhancement model can be used to handle common enhancement tasks, such as denoising, deblurring, etc., and the fine-tuning enhancement model can be set to perform targeted optimization according to specific needs, thereby making the video enhancement model easier to maintain and expand.

[0107] In some embodiments of the present disclosure, a video enhancement model is trained according to loss information and bit rate information to obtain a trained video enhancement model, including: training a fine-tuning enhancement module according to loss information and bit rate information to obtain a trained fine-tuning enhancement model; connecting a basic enhancement model and the trained fine-tuning enhancement model to obtain a trained video enhancement model.

[0108] Exemplarily, the loss information includes one or more of the following options: fidelity loss information, perception loss information, and image quality loss information.

[0109] The loss information is an important indicator for evaluating the performance of the video enhancement model and is used to guide the optimization direction of the video enhancement model during the training process. In an exemplary embodiment, the loss information may include one or more of fidelity loss information, perceptual loss information, and image quality loss information.

[0110] Fidelity loss information is mainly used to measure the pixel-level difference between the enhanced video frame and the original video frame. Among them, fidelity loss can include mean square error (MSE) or more advanced ones such as structural similarity (SSIM), etc., which aims to maintain a high degree of consistency in content between the enhanced video frame and the original video frame, that is, to minimize the distortion introduced by the enhancement process.

[0111] Perceptual loss focuses on the similarity of the enhanced video frames to the original video frames in terms of high-level semantics or visual perception. Specifically, a pre-trained deep learning model (such as the VGG network) is used to extract features of the video frames and compare the differences between these features. Perceptual loss helps the model generate visually more natural and realistic enhanced videos, even if the enhanced videos are not completely consistent with the original videos at the pixel level.

[0112] Image quality loss refers to the loss in improving the overall image quality of the video, which may include multiple factors such as clarity, color saturation, contrast, etc. In practical applications, image quality loss may be a comprehensive indicator that combines multiple specific loss functions to comprehensively evaluate and improve the image quality of the video.

[0113] In one possible implementation, after reconstructing the first processed video frame, the second processed video frame and the motion vector through a guideable encoder to obtain the reconstructed video frame and bit rate information, the fidelity loss information, the perceptual loss information and the image quality loss information between the reconstructed video frame and the second training video frame (original video frame) are calculated. Then, the fidelity loss information, the perceptual loss information, the image quality loss information and the bit rate information are combined into a total loss function according to certain weights, and the gradient of the loss function with respect to the model parameters is calculated using the back propagation algorithm. The parameters of the fine-tuning enhancement model are updated according to the gradient information to minimize the total loss function. Repeat the above process until the performance of the model is no longer significantly improved, or the predetermined training rounds are reached, and a trained fine-tuning enhancement model is obtained. Connect the basic enhancement model and the trained fine-tuning enhancement model to obtain a trained enhancement model.

[0114] In the disclosed embodiment, the video enhancement model is not limited to one model, but can be a basic enhancement model coupled with a fine-tuning enhancement model (i.e., a small model). During the model training process, the basic enhancement model does not participate in the training, and the fine-tuning enhancement model only needs to be trained, which can save training time, improve training speed, and reduce training costs. When new enhancement tasks or conditions need to be dealt with, only the fine-tuning enhancement model needs to be adjusted or replaced, which reduces the complexity and cost of model updates and improves the scalability of the model. In addition, by combining multiple loss information for training, it can be ensured that the model has good performance in multiple aspects, which helps to improve the overall performance of the video enhancement model.

[0115] Figure 4 A flowchart of training a video enhancement model according to an embodiment of the present disclosure is shown. Figure 4 Shown based on Figure 3 The process of training the video enhancement model with the network structure shown in Figure 4 , may include the following steps.

[0116] Step S401, obtaining a training video set; the training video set includes a plurality of training videos, the training videos include a plurality of training video frame pairs, and the training video frame pairs include a first adjacent training video frame and a second adjacent training video frame.

[0117] Step S402, degrade the first training video frame to obtain a first degraded video frame; input the first degraded video frame to the video enhancement model 310 for enhancement processing to obtain a first enhanced video frame; input the first enhanced video frame to the guideable video encoding pre-processing module 320 for video encoding pre-processing to obtain a first processed video frame.

[0118] Step S403, degrade the second training video frame to obtain a second degraded video frame; input the second degraded video frame to the video enhancement model 310 for enhancement processing to obtain a second enhanced video frame; input the second enhanced video frame to the guideable video encoding pre-processing module 320 for video encoding pre-processing to obtain a second processed video frame.

[0119] Step S404: Perform motion estimation on the first training video frame and the second training video frame through the motion estimation module 330 to obtain a motion vector of the second training video frame relative to the first training video frame.

[0120] Step S405: input the first processed video frame, the second processed video frame and the motion vector into the encoder 340 for reconstruction processing to obtain the reconstructed video frame and bit rate information.

[0121] Step S406: determining fidelity loss information, perception loss information and image quality loss information according to the reconstructed video frame and the second training video frame.

[0122] Step number 407, training the video enhancement model 310 according to the fidelity loss information, the perception loss information, the image quality loss information and the bit rate information to obtain a trained video enhancement model.

[0123] It should be noted that the training video set includes multiple training videos, the training videos include multiple training video frame pairs, the training video frame pairs include adjacent first training video frames and second training video frames, that is, the training video set includes multiple training video frame pairs, each training video frame pair includes adjacent first training video frames and second training video frames. Among them, step S402 to step S406 is the process of processing the first and second training video frames in each training video frame pair. After processing the first and second training video frames in the multiple training video frame pairs, the video enhancement model 310 is trained using the obtained loss information and bit rate information, and finally a trained video enhancement model is obtained.

[0124] In addition, the video enhancement model 310 may include a basic enhancement model and a fine-tuning enhancement model. During the model training process, the basic enhancement model does not participate in the training, and the fine-tuning enhancement model is continuously adjusted to obtain a trained fine-tuning enhancement model, which is then combined with the basic enhancement model to obtain a trained video enhancement model.

[0125] In the disclosed embodiment, by integrating video enhancement, pre-coding processing and coding process into a guideable framework, end-to-end optimization is achieved, and each link in the entire video processing flow is taken into account to ensure the maximization of overall performance. The video enhancement model takes into account the requirements of the coding process during the training process, so the enhanced video frames generated by the trained video enhancement model are easier to encode, which can reduce the bit rate required for coding while maintaining or improving the subjective image quality after coding. During the training process, the evaluation indicators such as fidelity loss, perceptual loss, and image quality loss are used to make the video enhancement model pay attention not only to the pixel-level error during the optimization process, but also to the visual quality of the video content, which helps to generate more natural and realistic video content; the bit rate information is introduced, and the model can learn how to maintain or improve the video quality while reducing the bit rate required for coding as much as possible during the training process, which helps to reduce bandwidth costs in practical applications. The guideable video coding pre-processing module can optimize the video content before coding, reduce redundant information, and reduce coding complexity. Due to the modular design, the system can flexibly adjust or replace each module to adapt to different application scenarios and needs.

[0126] Figure 5 A flow chart of a video enhancement method according to an embodiment of the present disclosure is shown. Figure 5 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The server 101 in the embodiment is also as follows Figure 1 The terminal device 103 in the embodiment is also as follows Figure 1 The server 101 and the terminal device 103 in the embodiment jointly implement the video enhancement method, but the present disclosure is not limited thereto. Figure 5 The video enhancement method provided by the embodiment of the present disclosure includes the following steps.

[0127] Step S510, obtaining a video to be processed; the video to be processed includes a plurality of video frames to be processed.

[0128] Step S520: input the video frame to be processed into the trained video enhancement model to obtain an enhanced video frame corresponding to the video frame to be processed.

[0129] The video to be processed is obtained, and the video frames to be processed contained in the video to be processed are input into the trained video enhancement model, and the video enhancement model is used to perform enhancement processing on the video frames to obtain enhanced video frames.

[0130] In the disclosed embodiment, during the process of training the video enhancement model, a guideable encoder is introduced, and the video encoding pre-processing (i.e., some smoothing or sharpening processing of the video frames before encoding) is simulated, so that the video enhancement model can be consistent with the reasoning process during the training process, and the video enhancement model is optimized end-to-end, so that the enhancement result of the trained video enhancement model is as friendly to the encoding process as possible, thereby achieving the effect of improving the subjective image quality after encoding and reducing the encoding bit rate.

[0131] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0132] Figure 6 FIG. 4 is a block diagram of a video enhancement model training device according to an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes a training video acquisition module 610, an enhancement processing module 620, a reconstruction processing module 630, a loss determination module 640 and a training module 650.

[0133] Among them, the training video acquisition module 610 is configured to: acquire a training video set; the training video set includes multiple training videos, the training videos include multiple training video frame pairs, and the training video frame pairs include adjacent first training video frames and second training video frames. The enhancement processing module 620 is configured to: enhance the first training video frame based on the video enhancement model to obtain a first processed video frame; enhance the second training video frame based on the video enhancement model to obtain a second processed video frame. The reconstruction processing module 630 is configured to: reconstruct the first processed video frame and the second processed video frame based on the guideable encoder to obtain a reconstructed video frame and bit rate information. The loss determination module 640 is configured to: determine the loss information based on the reconstructed video frame and the second training video frame. The training module 650 is configured to: train the video enhancement model based on the loss information and the bit rate information to obtain a trained video enhancement model.

[0134] In some embodiments of the present disclosure, the enhancement processing module 620 is further configured to: perform downgrading processing on the first training video frame to obtain a first degraded video frame; input the first degraded video frame into a video enhancement model for enhancement processing to obtain a first enhanced video frame; perform video encoding pre-processing on the first enhanced video frame to obtain a first processed video frame.

[0135] In some embodiments of the present disclosure, the enhancement processing module 620 is further configured to: perform downgrading processing on the second training video frame to obtain a second degraded video frame; input the second degraded video frame into the video enhancement model for enhancement processing to obtain a second enhanced video frame; perform video encoding pre-processing on the second enhanced video frame to obtain a second processed video frame.

[0136] In some embodiments of the present disclosure, the enhancement processing module 620 is further configured to perform pre-encoding processing on the enhanced video frame based on a guideable video pre-encoding processing module.

[0137] In some embodiments of the present disclosure, the apparatus 600 further includes a motion estimation module 660 configured to: perform motion estimation on the first training video frame and the second training video frame to obtain a motion vector of the second training video frame relative to the first training video frame.

[0138] In some embodiments of the present disclosure, the reconstruction processing module 630 is further configured to: input the first processed video frame, the second processed video frame and the motion vector into the guide encoder for reconstruction processing to obtain the reconstructed video frame and bit rate information.

[0139] In some embodiments of the present disclosure, the loss information includes one or more of the following options: fidelity loss information, perception loss information, and image quality loss information.

[0140] In some embodiments of the present disclosure, the video enhancement model includes a basic enhancement model and a fine-tuning enhancement model, the basic enhancement model is a pre-trained enhancement model, and the fine-tuning enhancement model is an enhancement model to be trained; wherein the training module 650 is further configured to: train the fine-tuning enhancement module according to loss information and bit rate information to obtain a trained fine-tuning enhancement model; connect the basic enhancement model and the trained fine-tuning enhancement model to obtain a trained video enhancement model.

[0141] Figure 7 FIG. 4 is a block diagram of a video enhancement device according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a to-be-processed video acquisition module 710 and a video enhancement module 720 .

[0142] The to-be-processed video acquisition module 710 is configured to: acquire the to-be-processed video; the to-be-processed video includes a plurality of to-be-processed video frames. The video enhancement module 720 is configured to: input the to-be-processed video frames into the trained video enhancement model to obtain enhanced video frames corresponding to the to-be-processed video frames.

[0143] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] Figure 8 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure. It should be noted that: Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0145] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0146] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 2 Follow the steps shown in .

[0147] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0148] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0149] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0150] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 840. As shown, the network adapter 840 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0151] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0152] The program product for implementing the above method according to the embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0153] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0154] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0156] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0157] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0158] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0159] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0161] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video enhancement model training method, characterized in that: The method comprises: Acquire a training video set; the training video set includes a plurality of training videos, the training videos include a plurality of training video frame pairs, and the training video frame pairs include adjacent first training video frames and second training video frames; Performing enhancement processing on the first training video frame based on the video enhancement model to obtain a first processed video frame; performing enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame; Reconstructing the first processed video frame and the second processed video frame based on a steerable encoder to obtain reconstructed video frames and bit rate information; Determining loss information according to the reconstructed video frame and the second training video frame; The video enhancement model is trained according to the loss information and the bit rate information to obtain a trained video enhancement model.

2. The method according to claim 1, characterized in that The step of performing enhancement processing on the first training video frame based on the video enhancement model to obtain a first processed video frame includes: Performing quality degradation on the first training video frame to obtain a first degraded video frame; Inputting the first degraded video frame into the video enhancement model for enhancement processing to obtain a first enhanced video frame; The first enhanced video frame is subjected to video encoding pre-processing to obtain the first processed video frame.

3. The method according to claim 1, characterized in that The step of performing enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame includes: Performing quality degradation on the second training video frame to obtain a second degraded video frame; Inputting the second degraded video frame into the video enhancement model for enhancement processing to obtain a second enhanced video frame; Perform video encoding pre-processing on the second enhanced video frame to obtain the second processed video frame.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: The enhanced video frames are pre-processed before encoding based on the guideable video encoding pre-processing module.

5. The training method according to claim 1, characterized in that: The method further comprises: Perform motion estimation on the first training video frame and the second training video frame to obtain a motion vector of the second training video frame relative to the first training video frame.

6. The method according to claim 5, characterized in that The reconstructing the first processed video frame and the second processed video frame based on the guideable encoder to obtain the reconstructed video frame and bit rate information includes: The first processed video frame, the second processed video frame and the motion vector are input into the guideable encoder for reconstruction processing to obtain the reconstructed video frame and the bit rate information.

7. The method according to claim 1, characterized in that The loss information includes one or more of the following options: fidelity loss information, perception loss information, and image quality loss information.

8. The method according to claim 1, characterized in that The video enhancement model includes a basic enhancement model and a fine-tuning enhancement model, wherein the basic enhancement model is a pre-trained enhancement model and the fine-tuning enhancement model is an enhancement model to be trained; The step of training the video enhancement model according to the loss information and the bit rate information to obtain the trained video enhancement model includes: Training the fine-tuning enhancement module according to the loss information and the bit rate information to obtain a trained fine-tuning enhancement model; The basic enhancement model and the trained fine-tuning enhancement model are connected to obtain the trained video enhancement model.

9. A video enhancement method, characterized in that: The method comprises: Acquire a video to be processed; the video to be processed includes a plurality of video frames to be processed; The video frame to be processed is input into a trained video enhancement model obtained by the training method of a video enhancement model according to any one of claims 1 to 8 to obtain an enhanced video frame corresponding to the video frame to be processed.

10. A video enhancement model training device, characterized in that: The device comprises: A training video acquisition module is configured to acquire a training video set; the training video set includes a plurality of training videos, the training videos include a plurality of training video frame pairs, and the training video frame pairs include adjacent first training video frames and second training video frames; an enhancement processing module, configured to perform enhancement processing on the first training video frame based on a video enhancement model to obtain a first processed video frame; and perform enhancement processing on the second training video frame based on the video enhancement model to obtain a second processed video frame; A reconstruction processing module, configured to reconstruct the first processed video frame and the second processed video frame based on a guideable encoder to obtain a reconstructed video frame and bit rate information; a loss determination module, configured to determine loss information according to the reconstructed video frame and the second training video frame; The training module is configured to train the video enhancement model according to the loss information and the bit rate information to obtain a trained video enhancement model.

11. A video enhancement device, characterized in that: The device comprises: A to-be-processed video acquisition module is configured to acquire a to-be-processed video; the to-be-processed video includes a plurality of to-be-processed video frames; The video enhancement module is configured to input the video frame to be processed into a trained video enhancement model obtained by the training method of the video enhancement model according to any one of claims 1 to 8, and obtain an enhanced video frame corresponding to the video frame to be processed.

12. An electronic device, characterized in that: include: processor; A memory for storing the processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the training method of the video enhancement model as described in any one of claims 1 to 8, or to implement the video enhancement method as described in claim 9.

13. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the training method of the video enhancement model as described in any one of claims 1 to 8, or implement the video enhancement method as described in claim 9.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the video enhancement model as described in any one of claims 1 to 8 is implemented, or the video enhancement method as described in claim 9 is implemented.

Images