Video stream processing method and device

Abstract

The embodiment of the invention provides a video stream processing method and device which are applied to the technical field of multimedia resource processing, and the method comprises the steps: obtaining any video frame of a video stream, carrying out the detection of the video frame, carrying out the super-resolution operation of a face image and the video frame after determining that the face image exists in the video frame, and carrying out the super-resolution operation of the face image and the video frame. Obtaining a high-resolution face image and a high-resolution video frame; and fusing the high-resolution face image and the high-resolution video frame to obtain a repaired video frame, and obtaining a repaired video stream according to each repaired video frame. According to the method and the device, the face part in the video is subjected to independent super-resolution processing, and the result of the super-resolution processing is fused with the super-resolution result of the video frame, so that the video containing complex degradation is repaired, and the high-quality super-resolution video stream is obtained. The details in the video stream are effectively recovered to improve the impression effect, the high definition of the whole video is kept, and the problems of fuzziness and detail missing in the prior art are solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of multimedia resource processing, and in particular to a method and device for processing a video stream. Background Art

[0002] With the rapid development of multimedia technology, video content has become an indispensable part of people's daily life and business activities. Video super-resolution is an important research direction in the field of image processing and computer vision, which aims to upgrade low-resolution videos to high-resolution videos.

[0003] Existing methods all perform super-resolution operations on the entire image of the video frame. Such methods are effective when processing natural scenes; however, in scenes containing a large number of real people, it is easy to cause blurring and unclearness of facial details. This is because these methods usually perform super-resolution processing on the entire image, but fail to process the details of the facial part in a targeted manner, resulting in particularly obvious detail loss in high-attention areas in the video (such as the face), and some unnatural textures or artifacts. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing a video stream, which are used to perform super-resolution operations on a face region detected in a video frame and the video frame respectively, so as to effectively restore the details of the face in the video.

[0005] In a first aspect, an embodiment of the present application provides a method for processing a video stream, including:

[0006] Get any video frame of the video stream;

[0007] After detecting the video frame and determining that a face image exists in the video frame, super-resolution operations are performed on the face image and the video frame respectively to obtain a high-resolution face image and a high-resolution video frame; and the high-resolution face image and the high-resolution video frame are fused to obtain a repaired video frame;

[0008] According to each repaired video frame, a repaired video stream is obtained.

[0009] This application performs separate super-resolution processing on high-interest areas in the video (such as the face) and merges the results with the super-resolution results of the video frame to repair the video containing complex degradation and obtain a high-quality super-resolution video stream. This can not only effectively restore the details in the video stream to improve the viewing effect, but also maintain the high definition of the overall video, solving the problems of blur and missing details in the existing technical solutions.

[0010] Optionally, performing a super-resolution operation on the facial image to obtain a high-resolution facial image includes:

[0011] Extracting multi-scale features of the face image;

[0012] Performing T times of denoising on the multi-scale features by a denoiser to obtain high-resolution image features;

[0013] Decoding the high-resolution image features to obtain a high-resolution face image;

[0014] Among them, each denoising process is as follows:

[0015] The kth noise and the multi-scale feature are input into the denoiser to obtain the k-1th noise, where k is any integer less than or equal to T, and the Tth noise is randomly generated.

[0016] Optionally, extracting multi-scale features of the face image includes:

[0017] Extracting shallow features of the face image;

[0018] The shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, wherein the high-frequency components are used to enhance image details and the low-frequency components are used to smooth the overall image.

[0019] Optionally, extracting shallow features of the face image includes:

[0020] Obtaining a shallow image of the face image through an encoder;

[0021] A k-th noise is added to the shallow image to obtain shallow features with primary degradation removed.

[0022] Optionally, the shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, including:

[0023] For any frequency component, the shallow features are processed by spatial feature transformation, multi-head self-attention mechanism and gated feedforward network to achieve feature extraction at the resolution corresponding to the frequency component.

[0024] Optionally, performing a super-resolution operation on the video frame to obtain a high-resolution video frame includes:

[0025] The generator removes the degradation phenomenon of the video frame to obtain a high-resolution video frame;

[0026] The generator is generated through training iteration based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result. The sample image is a low-resolution sample image generated after performing different degradation operations on the high-resolution sample image.

[0027] Optionally, fusing the high-resolution face image and the high-resolution video frame to obtain a repaired video frame includes:

[0028] Generating a mask for the high-resolution face image;

[0029] Controlling the weight distribution of the high-resolution face image and the high-resolution video frame according to the mask and fusing them to obtain a global image;

[0030] Perform tone matching processing on the global image to obtain a restored video frame.

[0031] This application performs separate super-resolution processing on high-interest areas in the video (such as the face) and merges the results with the super-resolution results of the video frame to repair the video containing complex degradation and obtain a high-quality super-resolution video stream. This can not only effectively restore the details in the video stream to improve the viewing effect, but also maintain the high definition of the overall video, solving the problems of blur and missing details in the existing technical solutions.

[0032] In a second aspect, an embodiment of the present application provides an interface testing device, including:

[0033] An acquisition module, used to acquire any video frame of a video stream;

[0034] A processing module is used to detect the video frame, determine that there is a face image in the video frame, perform super-resolution operation on the face image and the video frame respectively, to obtain a high-resolution face image and a high-resolution video frame; and fuse the high-resolution face image and the high-resolution video frame to obtain a repaired video frame;

[0035] The synthesis module is used to obtain a repaired video stream according to each repaired video frame.

[0036] This application performs separate super-resolution processing on high-interest areas in the video (such as the face) and merges the results with the super-resolution results of the video frame to repair the video containing complex degradation and obtain a high-quality super-resolution video stream. This can not only effectively restore the details in the video stream to improve the viewing effect, but also maintain the high definition of the overall video, solving the problems of blur and missing details in the existing technical solutions.

[0037] Optionally, the processing module is specifically used for:

[0038] The step of performing a super-resolution operation on the face image to obtain a high-resolution face image includes:

[0039] Extracting multi-scale features of the face image;

[0040] Performing T times of denoising on the multi-scale features by a denoiser to obtain high-resolution image features;

[0041] Decoding the high-resolution image features to obtain a high-resolution face image;

[0042] Among them, each denoising process is as follows:

[0043] The kth noise and the multi-scale feature are input into the denoiser to obtain the k-1th noise, where k is any integer less than or equal to T, and the Tth noise is randomly generated.

[0044] Optionally, the processing module is specifically used for:

[0045] Extracting shallow features of the face image;

[0046] The shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, wherein the high-frequency components are used to enhance image details and the low-frequency components are used to smooth the overall image.

[0047] Optionally, the processing module is specifically used for:

[0048] Obtaining a shallow image of the face image through an encoder;

[0049] A k-th noise is added to the shallow image to obtain shallow features with primary degradation removed.

[0050] Optionally, the processing module is specifically used for:

[0051] For any frequency component, the shallow features are processed by spatial feature transformation, multi-head self-attention mechanism and gated feedforward network to achieve feature extraction at the resolution corresponding to the frequency component.

[0052] Optionally, the processing module is specifically used for:

[0053] The generator removes the degradation phenomenon of the video frame to obtain a high-resolution video frame;

[0054] The generator is generated through training iteration based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result. The sample image is a low-resolution sample image generated after performing different degradation operations on the high-resolution sample image.

[0055] Optionally, the processing module is specifically used for:

[0056] Generating a mask for the high-resolution face image;

[0057] Controlling the weight distribution of the high-resolution face image and the high-resolution video frame according to the mask and fusing them to obtain a global image;

[0058] Perform tone matching processing on the global image to obtain a restored video frame.

[0059] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any of the video stream processing methods described in the first aspect.

[0060] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer device executes any of the video stream processing methods described in the first aspect above.

[0061] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it implements any video stream processing method described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0063] Figure 1 A schematic diagram of a process flow of a video stream processing method provided in an embodiment of the present application;

[0064] Figure 2 A schematic diagram of a video stream processing system architecture provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of a face super-resolution model provided in an embodiment of the present application performing super-resolution operation on a face image;

[0066] Figure 4 A schematic diagram of a process of processing shallow features by a converter block provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of a video stream processing device provided in an embodiment of the present application;

[0068] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0070] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0071] Generative Adversarial Networks (GAN): Generative Adversarial Networks consist of a generative model and a discriminative model. The generative model is responsible for capturing the distribution of sample data, while the discriminative model is generally a binary classifier that determines whether the input is real data or a generated sample. The optimization process of this model is a "binary minimax game" problem. During training, one of the parties (the discriminative network or the generative network) is fixed, and the parameters of the other model are updated, and the iterations are repeated. Finally, the generative model can estimate the distribution of sample data.

[0072] Super-resolution (SR) refers to the process of using optics and related optical knowledge to restore image details and other data information based on known image information. In simple terms, it increases the resolution of the image to prevent its image quality from deteriorating.

[0073] Figure 1 The following is an exemplary process of a method for processing a video stream, which specifically includes the following steps:

[0074] Step S101, obtaining any video frame of a video stream.

[0075] Specifically, after acquiring the video stream, the video frame can be preprocessed using advanced image analysis algorithms, for example, by removing possible impulse noise through median filtering, and then using histogram equalization to adjust brightness and contrast to make the video frame clearer.

[0076] Step S102, detecting the video frame, and after determining that a face image exists in the video frame, performing super-resolution operations on the face image and the video frame respectively to obtain a high-resolution face image and a high-resolution video frame; and fusing the high-resolution face image and the high-resolution video frame to obtain a repaired video frame.

[0077] Specifically, an efficient face detection algorithm, such as a face detection model based on deep learning, can be used to detect the preprocessed video frame. If it is determined that there is a face image in the video frame, the face area is cropped, and different super-resolution algorithms are used to process the face image and the video frame respectively to obtain a high-resolution face image and a high-resolution video frame. If it is determined that there is no face image in the video frame, the video frame is directly processed by super-resolution to obtain a high-resolution video frame. In the fusion process of the high-resolution face image and the high-resolution video frame, the position, size and importance of the face image can be used to reasonably adjust the fusion weight with the video frame to ensure that the fused image is natural and harmonious. Among them, the face detection algorithm model can use lightweight models such as RetinaFace and MTCNN. If there are multiple face images in the video frame, each face image is subjected to super-resolution operation to obtain multiple high-resolution face images, which are then fused with the high-resolution video frame obtained after the super-resolution operation.

[0078] In some other embodiments, before detecting the video frame, the quality of the video frame can be evaluated by calculating the clarity index, signal-to-noise ratio and other parameters of the video frame. When the quality is lower than a certain threshold, the subsequent face detection and super-resolution operations are triggered, which can effectively improve the processing efficiency and avoid unnecessary processing of high-quality video frames.

[0079] Step S103, obtaining a repaired video stream according to each repaired video frame.

[0080] Specifically, the obtained repaired video frames are combined in sequence to form a repaired video stream.

[0081] This application performs separate super-resolution processing on high-interest areas in the video (such as the face) and merges the results with the super-resolution results of the video frame to repair the video containing complex degradation and obtain a high-quality super-resolution video stream. This can not only effectively restore the details in the video stream to improve the viewing effect, but also maintain the high definition of the overall video, solving the problems of blur and missing details in the existing technical solutions.

[0082] like Figure 2As shown, a video stream processing system architecture 200 provided by the present application includes: a face detection module 201, a face super-resolution model 202, a video frame super-resolution model 203 and a fusion module 204. Among them, the face detection module 201 is used to detect whether a video frame contains a face image; the face super-resolution model 202 is used to perform a super-resolution operation on the face image to obtain a high-resolution face image; the video frame super-resolution model 203 is used to perform a super-resolution operation on the video frame to obtain a high-resolution video frame; the fusion module 204 is used to fuse the high-resolution face image and the high-resolution video frame to obtain a repaired video frame.

[0083] In the above step S102, a super-resolution operation is performed on the face image to obtain a high-resolution face image, including: extracting multi-scale features of the face image; denoising the multi-scale features T times through a denoiser to obtain high-resolution image features; decoding the high-resolution image features to obtain a high-resolution face image; wherein each denoising process is as follows: inputting the kth noise and the multi-scale features into the denoiser to obtain the k-1th noise, k is any integer less than or equal to T, and the Tth noise is randomly generated.

[0084] Specifically, the face super-resolution model is used to perform super-resolution operation on the face image. The face super-resolution model includes a denoiser, which is used to denoise the multi-scale features extracted from the face image T times, where T is a preset diffusion time step. The high-resolution image features obtained after T times of denoising are decoded to obtain a high-resolution face image. For the first denoising process, Gaussian noise can be used to randomly initialize the Tth noise, denoted as x T ; x T And multi-scale features are input to the denoiser, and the T-1th noise can be obtained, denoted as x T-1 ; The second denoising process is to use the T-1th noise x T-1 And multi-scale features are input to the denoiser, and the T-2th noise can be obtained, denoted as x T-2 ; And so on, each denoising process will use the denoising result of the previous denoising process until the diffusion time step T is 0, and the noise x0 is obtained. The noise x0 is decoded to obtain the high-resolution image features, and the high-resolution face image is obtained based on the high-resolution image features.

[0085] In one embodiment, multi-scale features of a face image may be extracted in the following manner: shallow features of the face image are extracted; the shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, wherein high-frequency components are used to enhance image details and low-frequency components are used to smooth the overall image.

[0086] Specifically, the face super-resolution model also includes a degradation removal module and a multi-scale feature extraction module. The degradation removal module can focus on the key areas in the face image by introducing a feature selection mechanism (using an attention mechanism or weighted average). For example, using an encoder based on an attention mechanism, in the process of extracting shallow features, the attention distribution of the face image is automatically learned, and higher weights are given to key areas such as the eyes, mouth, and nose. A combination of a spatial attention mechanism and a channel attention mechanism can be used. The spatial attention mechanism focuses on different positions in the image, and the channel attention mechanism focuses on different feature channels. In this way, the key features of the face image can be extracted more effectively. This selective fusion process ensures that the important features in the face image are strengthened, while irrelevant or redundant features are suppressed, thereby improving the quality of the shallow feature representation of the face image.

[0087] The multi-scale feature extraction module can decompose the face image into multiple frequency components, such as high-frequency components and low-frequency components, through filters or convolution operations, and separate different frequency information for independent processing. For high-frequency components, a deep convolutional neural network (CNN) can be used to enhance image details. For example, a CNN architecture with multiple convolutional layers and pooling layers is designed to continuously extract local features and edge information of the image to enhance the expressiveness of high-frequency components. For low-frequency components, a recurrent neural network (RNN) can be used to smooth the entire image. RNN can effectively process sequence data. For the low-frequency components of the image, it can capture the overall structure and trend of the image, perform smoothing, and reduce noise and interference. At the same time, CNN and RNN can be combined to form a hybrid neural network structure to give full play to the advantages of both and improve the extraction effect of multi-scale features. The multi-scale feature extraction module can also dynamically adjust the weights of different frequency components according to the type and scene of the video stream. For example, in a close-up video stream of a person, the weight of the high-frequency component is increased to highlight the details of the face, such as skin texture, hair, etc. The type and scene of the video can be automatically identified by analyzing the content characteristics of the video stream, and then the weights of different frequency components can be adjusted according to preset rules. For landscape video streams, the weight of high-frequency components can be appropriately reduced and the weight of low-frequency components can be increased to highlight the overall picture beauty and color coordination. At the same time, a personalized frequency component weight adjustment function can be provided based on user preferences and needs to meet the viewing needs of different users.

[0088] like Figure 3 As shown in the figure, it is a flow chart of a face super-resolution model proposed in the present application for super-resolution operation on a face image. The face super-resolution model performs T denoising on a low-resolution face image. The first denoising process is: removing the degradation module to obtain a shallow image of the low-resolution face image through the encoder network, and then adding the shallow image to the random initialization to obtain the Tth noise xT , reduce shallow layer degradation to obtain shallow layer features, the multi-scale feature extraction module processes the shallow layer features according to different frequency components to obtain multi-scale features at different resolutions, and adds the multi-scale features to the denoiser to guide the denoising process to obtain the T-1th noise x T-1 The second denoising process is: remove the degradation module to obtain the shallow image of the low-resolution face image through the encoder network, and then add the shallow image to the T-1th noise x T-1 , reduce shallow layer degradation to obtain shallow layer features, the multi-scale feature extraction module processes the shallow layer features according to different frequency components to obtain multi-scale features at different resolutions, and adds the multi-scale features to the denoiser to guide the denoising process to obtain the T-2 noise x T-2 , and so on until the diffusion time step T is 0, the noise x0 is obtained, the noise x0 is decoded to obtain the high-resolution image features, and the high-resolution face image is obtained according to the high-resolution image features.

[0089] In one embodiment, shallow features of a face image may be extracted in the following manner: obtaining a shallow image of the face image through an encoder; adding k-th noise to the shallow image to obtain shallow features with primary degradation removed.

[0090] Specifically, by removing the encoder in the degradation module, the 512×512 pixel image is encoded into a 64×64 shallow image, and the kth noise is added to the shallow image to extract the shallow features of the face image. The extraction process can use multiple encoders for parallel processing, and then fuse to obtain richer shallow features. For example, multiple encoders with different structures are designed, such as CNN-based encoders, autoencoder-based encoders, etc., to encode face images respectively. Each encoder can extract shallow features from different aspects, and then these features are fused through the feature fusion layer.

[0091] In addition, a combination of multiple methods can be used to remove primary degradation. In addition to adding noise, image enhancement, edge detection and other operations can also be performed. For example, image enhancement methods such as histogram equalization and contrast enhancement can be used to improve the contrast and brightness of facial images and reduce the impact of primary degradation. At the same time, edge detection algorithms such as Sobel operator and Canny operator are used to detect the edge information of facial images, and then the edges are enhanced to improve the edge clarity of shallow features. According to the quality and degradation degree of the facial image, the appropriate primary degradation removal method can be automatically selected for combination to improve the quality of shallow features.

[0092] Optionally, the multi-scale feature extraction module processes shallow features according to different frequency components to obtain multi-scale features at different resolutions, including: for any frequency component, processing the shallow features through spatial feature transformation, multi-head self-attention mechanism and gated feedforward network to achieve feature extraction at the resolution corresponding to the frequency component.

[0093] Specifically, the processing of high-frequency components usually focuses on the enhancement of details and edges, while the processing of low-frequency components focuses on the smoothing and color adjustment of the overall image. Different spatial feature transformation methods can be used for the processing of different frequency components. For low-frequency components, linear transformations such as matrix multiplication and translation can be used to smooth shallow features and highlight the overall structure of the image. For high-frequency components, linear transformations such as wavelet transform and Fourier transform can be used to extract image detail information and edge features. According to the characteristics of different frequency components and processing requirements, appropriate spatial feature transformation methods can be selected for combination. For example, linear transformation is first performed on low-frequency components, and then Gaussian blur processing is performed to further smooth the image; for high-frequency components, wavelet transform is first performed, and then enhancement processing is performed to highlight details. In the multi-head self-attention mechanism, the weight distribution of the attention head can be adjusted according to different frequency components. For example, for high-frequency components, the attention head can pay more attention to edge information; for low-frequency components, the attention head can pay more attention to the overall structure of the image. In this way, features under different frequency components can be extracted more effectively. The gated feedforward network can use different activation functions and gating mechanisms to adapt to the processing requirements of different frequency components. For high-frequency components, the ReLU activation function can be used to enhance the nonlinear expression ability of the network and highlight the details of the image. For low-frequency components, the Sigmoid activation function can be used to achieve smooth control of features and highlight the overall structure of the image.

[0094] The multi-scale feature extraction module structure contains several transformer blocks, each of which processes a frequency component of the shallow features of the face image, such as Figure 4 As shown in the figure, a flowchart of each transformer block in the multi-scale feature extraction module in this embodiment processes a frequency component of the shallow feature, and the shallow feature is defined as F in , modulate the shallow feature F1 through spatial feature transformation to obtain F2; capture the shallow feature F through the multi-head self-attention mechanism in The global and local context information is obtained to obtain F3; finally, F3 is processed by the gated feedforward network to obtain the output feature F out .

[0095] In the above step S102, a super-resolution operation can be performed on the video frame to obtain a high-resolution video frame in the following manner: a generator is used to remove the degradation phenomenon of the video frame to obtain a high-resolution video frame; the generator is iteratively generated based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result, and the sample image is a low-resolution sample image generated after different degradation operations are performed on the high-resolution sample image.

[0096] Generative Adversarial Networks (GAN) can be used to perform super-resolution operations on video frames, where the generative adversarial network includes a generative model and a discriminative model. The following is a detailed introduction to the process of training the generator. The generator to be trained is used to remove the degradation phenomenon of the low-resolution sample image to obtain a high-resolution video frame. The high-resolution image generated by the generator will be passed to the discriminator. The discriminator fuses features of different scales through jump connections and outputs a authenticity score for each pixel to evaluate the authenticity of the generated image. The generator adjusts its parameters based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result until the generator reaches the training iteration termination condition, and the generator training is completed. Among them, the discriminator can be a discriminator of the U-Net structure.

[0097] When preparing sample images, most existing methods artificially simulate fixed degradation factors, such as Gaussian blur, motion blur, scaling, etc. These limited and single degradation methods cannot simulate the diverse and complex degradation processes in real scenes. Therefore, in the process of training the generator, the sample images can be subjected to repeated classic degradation operations. Each degradation operation may involve motion blur, downsampling, noise, and image compression. For example, for the existing sample images a, b, and c, Gaussian blur, motion blur, scaling, and other operations are performed on sample image a, motion blur, scaling, Gaussian blur, and other operations are performed on sample image b, and scaling, motion blur, Gaussian blur, and other operations are performed on sample image c. Different degradation operations can also be performed on different sample images. For example, Gaussian blur is performed on sample image a, scaling is performed on sample image b, and motion blur is performed on sample image c.

[0098] In the above step S103, the high-resolution face image and the high-resolution video frame can be fused to obtain a repaired video frame in the following manner: generating a mask for the high-resolution face image; controlling the weight distribution of the high-resolution face image and the high-resolution video frame according to the mask and fusing them to obtain a global image; performing tone matching processing on the global image to obtain a repaired video frame.

[0099] Specifically, to ensure the natural transition of the fusion process of high-resolution face images and high-resolution video frames, a mask generator based on a convolutional neural network can be designed. The high-resolution face image is input and a gradually blurred mask is generated for the high-resolution face image. This mask can ensure that the transition part of the face edge is smooth and natural, avoiding abrupt visual effects. The high-resolution face image and the high-resolution video frame are fused at the pixel level by the weighted average method, and the weight distribution of the high-resolution face image and the high-resolution video frame is controlled by the mask, so that the transition between the face and the background is smooth and natural. In the fusion process, different fusion methods can be used according to the weight distribution, such as weighted average fusion, fusion based on the attention mechanism, etc., to obtain better fusion effect. After the fusion is completed, a global hue matching process is performed to ensure that the color of the face area is consistent with the background area, further improving the overall visual consistency of the image. In the hue matching process, multiple color spaces can be used for conversion and matching, such as RGB, HSV, etc., to obtain better hue consistency. For example, in the RGB color space, the color value of each channel can be adjusted to make the fused image more coordinated in color. In the HSV color space, you can adjust parameters such as hue, saturation, and brightness to make the image more natural. At the same time, you can adjust the hue according to the style and needs of the video stream, such as increasing contrast and saturation, to enhance the visual effect of the image. You can use a hue adjustment method based on deep learning to allow the network to automatically learn hue adjustment parameters under different styles and needs, and improve the intelligence level of hue matching processing.

[0100] like Figure 5 As shown, a video stream processing device 500 is provided for an embodiment of the present application, including:

[0101] An acquisition module 501 is used to acquire any video frame of a video stream;

[0102] The processing module 502 is used to detect the video frame, determine that there is a face image in the video frame, perform super-resolution operation on the face image and the video frame respectively, to obtain a high-resolution face image and a high-resolution video frame; and fuse the high-resolution face image and the high-resolution video frame to obtain a repaired video frame;

[0103] The synthesis module 503 is used to obtain a restored video stream according to each restored video frame.

[0104] This application performs separate super-resolution processing on high-interest areas in the video (such as the face) and merges the results with the super-resolution results of the video frame to repair the video containing complex degradation and obtain a high-quality super-resolution video stream. This can not only effectively restore the details in the video stream to improve the viewing effect, but also maintain the high definition of the overall video, solving the problems of blur and missing details in the existing technical solutions.

[0105] Optionally, the processing module 502 is specifically configured to:

[0106] The step of performing a super-resolution operation on the face image to obtain a high-resolution face image includes:

[0107] Extracting multi-scale features of the face image;

[0108] Performing T times of denoising on the multi-scale features by a denoiser to obtain high-resolution image features;

[0109] Decoding the high-resolution image features to obtain a high-resolution face image;

[0110] Among them, each denoising process is as follows:

[0111] The kth noise and the multi-scale feature are input into the denoiser to obtain the k-1th noise, where k is any integer less than or equal to T, and the Tth noise is randomly generated.

[0112] Optionally, the processing module 502 is specifically configured to:

[0113] Extracting shallow features of the face image;

[0114] The shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, wherein the high-frequency components are used to enhance image details and the low-frequency components are used to smooth the overall image.

[0115] Optionally, the processing module 502 is specifically configured to:

[0116] Obtaining a shallow image of the face image through an encoder;

[0117] A k-th noise is added to the shallow image to obtain shallow features with primary degradation removed.

[0118] Optionally, the processing module 502 is specifically configured to:

[0119] For any frequency component, the shallow features are processed by spatial feature transformation, multi-head self-attention mechanism and gated feedforward network to achieve feature extraction at the resolution corresponding to the frequency component.

[0120] Optionally, the processing module 502 is specifically configured to:

[0121] The generator removes the degradation phenomenon of the video frame to obtain a high-resolution video frame;

[0122] The generator is generated through training iteration based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result. The sample image is a low-resolution sample image generated after performing different degradation operations on the high-resolution sample image.

[0123] Optionally, the processing module 502 is specifically configured to:

[0124] Generating a mask for the high-resolution face image;

[0125] Controlling the weight distribution of the high-resolution face image and the high-resolution video frame according to the mask and fusing them to obtain a global image;

[0126] Perform tone matching processing on the global image to obtain a restored video frame.

[0127] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 6 The terminal device 601 and / or the server 602 shown in FIG. Figure 6 As shown, it includes at least one processor 601 and a memory 602 connected to the at least one processor. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6 For example, the processor 601 and the memory 602 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0128] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601, and the at least one processor 601 can perform the steps of the above-mentioned video stream processing method by executing the instructions stored in the memory 602.

[0129] Among them, the processor 601 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device. By running or executing instructions stored in the memory 602 and calling data stored in the memory 602, the low-resolution video frames are processed to obtain high-resolution video frames. Optionally, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0130] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0131] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0132] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned video stream processing method.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0138] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing a video stream, characterized in that: include: Get any video frame of the video stream; After detecting the video frame and determining that a face image exists in the video frame, super-resolution operations are performed on the face image and the video frame respectively to obtain a high-resolution face image and a high-resolution video frame; and the high-resolution face image and the high-resolution video frame are fused to obtain a repaired video frame; According to each repaired video frame, a repaired video stream is obtained.

2. The method according to claim 1, characterized in that The step of performing a super-resolution operation on the face image to obtain a high-resolution face image includes: Extracting multi-scale features of the face image; Performing T times of denoising on the multi-scale features by a denoiser to obtain high-resolution image features; Decoding the high-resolution image features to obtain a high-resolution face image; Among them, each denoising process is as follows: The kth noise and the multi-scale feature are input into the denoiser to obtain the k-1th noise, where k is any integer less than or equal to T, and the Tth noise is randomly generated.

3. The method according to claim 2, characterized in that The extracting multi-scale features of the face image comprises: Extracting shallow features of the face image; The shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, wherein the high-frequency components are used to enhance image details and the low-frequency components are used to smooth the overall image.

4. The method according to claim 3, characterized in that The extracting of shallow features of the face image comprises: Obtaining a shallow image of the face image through an encoder; A k-th noise is added to the shallow image to obtain shallow features with primary degradation removed.

5. The method according to claim 3, characterized in that The shallow features are processed according to different frequency components to obtain multi-scale features at different resolutions, including: For any frequency component, the shallow features are processed by spatial feature transformation, multi-head self-attention mechanism and gated feedforward network to achieve feature extraction at the resolution corresponding to the frequency component.

6. The method according to any one of claims 1 to 5, characterized in that: The performing a super-resolution operation on the video frame to obtain a high-resolution video frame includes: The generator removes the degradation phenomenon of the video frame to obtain a high-resolution video frame; The generator is generated through training iteration based on the restoration result of the sample image and the evaluation result of the discriminator on the restoration result. The sample image is a low-resolution sample image generated after performing different degradation operations on the high-resolution sample image.

7. The method according to any one of claims 1 to 5, characterized in that: The fusing the high-resolution face image and the high-resolution video frame to obtain a repaired video frame includes: generating a mask for the high-resolution face image; Controlling the weight distribution of the high-resolution face image and the high-resolution video frame according to the mask and fusing them to obtain a global image; Perform tone matching processing on the global image to obtain a restored video frame.

8. A video stream processing device, characterized in that: include: An acquisition module, used to acquire any video frame of a video stream; A processing module is used to detect the video frame, determine that there is a face image in the video frame, perform super-resolution operation on the face image and the video frame respectively, to obtain a high-resolution face image and a high-resolution video frame; and fuse the high-resolution face image and the high-resolution video frame to obtain a repaired video frame; The synthesis module is used to obtain a repaired video stream according to each repaired video frame.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of any one of the methods of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 7.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of any method described in claims 1 to 7 are implemented.

Images