Video restoration and enhancement method for adaptive image processing

Abstract

The invention discloses a video restoration and enhancement method for adaptive image processing. The method comprises the following steps: carrying out noise type identification on each frame of image in a target video; the types comprise vertical bar noise, snowflake noise and image flicker; if the vertical bar noise exists in the image, a denoising model based on a convolutional neural network is called to denoise the vertical bar noise; if the snowflake noise exists in the image, calling a denoising model based on a generative adversarial network to denoise the snowflake noise; if image flicker exists in the image, a time sequence stabilization algorithm is called to denoise the image flicker; and carrying out image quality enhancement on the denoised image by using a super-resolution model. The method can improve the restoration effect and the enhancement effect, and improves the calculation speed at the same time.

Description

Technical Field

[0001] This application relates to the field of image restoration methods and enhancement technologies, and particularly to a video restoration and enhancement method for adaptive image processing. Background Art

[0002] Currently, video restoration and super-resolution enhancement technologies have become research hotspots in the field of video processing and are widely used in multiple scenarios such as old video restoration, video quality improvement, and video forgery detection. With the popularity of short videos and high-definition videos, people's requirements for video quality are getting higher and higher, and the importance of video restoration and super-resolution enhancement technologies has become even more prominent. These technologies aim to repair damaged areas in videos, remove noise, improve resolution, and enhance the visual effects of videos through algorithms.

[0003] However, existing video restoration and super-resolution enhancement technologies still face many challenges. Especially when dealing with old videos and high-noise videos with severe damage, existing restoration methods can no longer meet the requirements of high quality and large scale. Existing video restoration methods usually rely on template-based image restoration, filtering algorithms, local algorithms, etc.; filtering algorithms mainly remove noise by smoothing image frames, but these methods often cause detail blurring and distortion when dealing with some high-noise and damaged areas. Local restoration algorithms fill damaged areas by copying surrounding pixels, but this method is prone to problems such as unnatural or incoherent video content. Moreover, these methods can remove some simple noises, but have poor effects on the restoration of complex noises (such as snow noise, stripe noise) and high-frequency detail loss. Usually, a large amount of computing resources are required, and when dealing with high-resolution videos, high time and computing costs are often encountered. Traditional super-resolution technologies mainly rely on interpolation methods such as bilinear interpolation and cubic interpolation, and these methods usually cannot effectively restore lost details and may even cause a decrease in image resolution. Summary of the Invention

[0004] Based on this, in view of the above technical problems, a video restoration and enhancement method for adaptive image processing is provided to solve the problems that the existing technology has poor restoration effects and traditional super-resolution technologies are prone to losing details.

[0005] In a first aspect, a video restoration and enhancement method for adaptive image processing, the method includes:

[0006] Construct a denoising model based on a convolutional neural network; the denoising model based on the convolutional neural network is composed of a convblock block, a dilated convolutional kernel, a residuai Block, a noise prediction, and a noise removal module to predict noise according to the extracted features, and then output a denoised image by subtracting the predicted noise from the original image;

[0007] Construct a denoising model based on a generative adversarial network; the generator in the denoising model based on the generative adversarial network consists of multiple convolutional layers, multiple residual modules, and one spatial channel attention. The residual module contains two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules, and one spatial channel attention adopt a symmetric structure with skip connections;

[0008] Construct a hybrid model of a convolutional network and an adversarial network; the hybrid model of a convolutional network and an adversarial network consists of a convolutional module and a generative adversarial network module;

[0009] Construct a super-resolution model for deep learning; the super-resolution model for deep learning includes a shallow network, a deep network, and adaptive convolution for adaptively restoring the edges and texture details of an image; the super-resolution model for deep learning is optimized using multiple loss functions;

[0010] Receive a target video and identify the noise type for each frame image in the target video; the noise types include vertical bar noise, snowflake noise, image flickering, and other specific noises;

[0011] Perform noise removal on each frame image, specifically including: if vertical bar noise exists in the image, call the denoising model based on a convolutional neural network to denoise the vertical bar noise; if snowflake noise exists in the image, call the denoising model based on a generative adversarial network to denoise the snowflake noise; if image flickering exists in the image, call the temporal stabilization algorithm to denoise the image flickering; if other specific noises exist in the image, call the hybrid model of a convolutional network and an adversarial network to denoise;

[0012] Use the super-resolution model for deep learning to enhance the image quality of the denoised image.

[0013] In the above solution, optionally, after calling the denoising model based on a convolutional neural network, the denoising model based on a generative adversarial network, and the hybrid model of a convolutional network and an adversarial network, it further includes: determining whether the denoised image flickers, and if so, using the temporal front and back smoothing algorithm to repair the flickering.

[0014] In the above solution, optionally, the process of the temporal stabilization algorithm includes: analyzing the brightness fluctuations of the front and back frames through temporal analysis, calculating the brightness change of each pixel to identify image flickering; using image segmentation technology to divide the detail area or the background area; for the image flickering in the detail area, dynamically adjusting the weight according to the temporal change, and smoothing and eliminating the fluctuation through the weighted median value; for the image flickering in the background area, the median is used to eliminate the fluctuation.

[0015] In the above solution, further optionally, the calculation of the brightness change of each pixel is implemented by the following formula:

[0016]

[0017] where L t (x, y) is the brightness value of the current frame, and ΔL(x, y, t) is the brightness change.

[0018] In the above solution, further optionally, the elimination of fluctuations is performed by the following formula:

[0019]

[0020] where W t is the weight.

[0021] In the above solution, optionally, the super-resolution model of deep learning is specifically implemented by the following formula:

[0022] The shallow network is implemented by the following formula:

[0023] Feature extraction is performed using a small convolution kernel;

[0024] F1(x, y) = σ(W1 * I low (x, y) + b1)

[0025] where I low (x, y) is the input low-resolution image, F1(x, y) is the output feature map, W1 is the convolution kernel, b1 is the bias, and σ is the activation function;

[0026] Dimensionality reduction is achieved through a pooling layer, and the max-pooling layer is implemented by the following formula:

[0027] P(x, y) = max(F1(x, y))

[0028] where F1(x, y) is the value of the input feature map at the position (x, y), P(x, y) is the value of the output feature map at the position (x, y), and the max(·) operation represents taking the maximum value of all values within the pooling window;

[0029] For the deep network, a residual convolution block is introduced:

[0030] F2(x, y) = F1(x, y) + R(F1(x, y))

[0031] where R represents the residual module, which can alleviate the problem of gradient disappearance and enhance the transmission of detailed features using cross-layer skip connections;

[0032] The adaptive convolution is constructed by the following formula:

[0033] W adaptive = f(region, featuremap)

[0034] Among them, the weight generation function W is a function for dynamically generating the weights of the convolutional kernel, f is a learnable function for dynamically generating the weights of the convolutional kernel according to the region type and the feature map, region represents the region type processed by the current convolutional operation, and feature map represents the input feature map;

[0035] Generate the fine-tuned image:

[0036] I enhances (x, y) = W adaptive * F2(x, y)

[0037] I enhances (x, y) is the result of the adjusted image.

[0038] In the above solution, optionally, the optimization using the multiple loss functions includes:

[0039] The final loss function is obtained by combining the multiple loss functions through the following formula:

[0040] L total = αL perceptual + βL GAN + γL MSE

[0041] Among them, α, β, γ are weight hyperparameters, L perceptual is the perceptual loss function, L GAN is the adversarial loss function, L MSE is the mean squared error loss function.

[0042] In a second aspect, a video restoration and enhancement system for adaptive image processing, the system includes:

[0043] A denoising model construction module based on a convolutional neural network: used to construct a denoising model based on a convolutional neural network; the denoising model based on a convolutional neural network is composed of a conv block, a dilated convolutional kernel, a residuai Block, a noise prediction, and a noise removal module, to predict the noise according to the extracted features, and then output the denoised image by subtracting the predicted noise from the original image;

[0044] Denoising model construction module based on generative adversarial network: used to construct a denoising model based on generative adversarial network; the generator in the denoising model based on generative adversarial network consists of multiple convolutional layers, multiple residual modules and one spatial channel attention, and the residual module contains two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules and one spatial channel attention adopt a symmetric structure of skip connection;

[0045] Hybrid model construction module based on convolutional network and adversarial network: used to construct a hybrid model based on convolutional network and adversarial network; the hybrid model based on convolutional network and adversarial network consists of a convolutional module and a generative adversarial network module;

[0046] Super-resolution model construction module for deep learning: used to construct a super-resolution model for deep learning; the super-resolution model for deep learning includes a shallow network, a deep network and adaptive convolution, which are used to adaptively restore the edges and texture details of images; the super-resolution model for deep learning is optimized using multiple loss functions;

[0047] Noise identification module: used to receive a target video and identify the noise type for each frame image in the target video; the noise types include vertical bar noise, snowflake noise, image flicker, and other specific noises;

[0048] Noise removal module: used to remove noise from each frame image, specifically including: if there is vertical bar noise in the image, call the denoising model based on convolutional neural network to denoise the vertical bar noise; if there is snowflake noise in the image, call the denoising model based on generative adversarial network to denoise the snowflake noise; if there is image flicker in the image, call the temporal stabilization algorithm to denoise the image flicker; if there is other specific noise in the image, call the hybrid model based on convolutional network and adversarial network to denoise;

[0049] Image quality enhancement module: used to enhance the image quality of the denoised image using the super-resolution model for deep learning.

[0050] In a third aspect, a computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method for video repair and enhancement of adaptive image processing described in the first aspect above.

[0051] In a fourth aspect, a computer program product includes computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the method for video repair and enhancement of adaptive image processing described in the first aspect above.

[0052] This application has at least the following beneficial effects:

[0053] This application classifies the noise in the image and uses the corresponding constructed deep learning noise model to remove it. For example, for vertical bar noise, a denoising model based on a convolutional neural network is called to denoise the vertical bar noise; in the denoising model based on a convolutional neural network, by introducing shallow feature extraction and multi-scale processing, the model can capture more image details, and the noise distribution is explicitly modeled through a noise prediction module to improve the denoising accuracy. A denoising model based on a generative adversarial network is called to denoise snowflake noise; in the denoising model based on a generative adversarial network for snowflake noise, by stacking residual modules, spatial-channel attention modules, etc. in the generator network, each residual block contains two convolutional layers, two batch normalizations, and two ReLU activation functions to increase the depth of the network. A symmetric skip connection structure is also included in the network to make the network have better convergence. The skip connection of the residual module is input to the spatial-channel attention module to further extract important spatial features such as edges and textures in the feature map and suppress unimportant background regions or noise regions. The generative adversarial network model combining the above advantages can improve the denoising effect of the picture and is suitable for denoising tasks with texture features such as snowflake noise. Other noises are processed by a hybrid model of a convolutional network and an adversarial network, where the transposed convolution in the convolutional module can learn more complex mapping relationships. This helps to restore the detail information of the image. The residual block can enhance the expressive ability of the model and avoid over-smoothing. Combining the above characteristics, this model can process various types of noises (such as Gaussian noise, salt-and-pepper noise, real noise, etc.). For image flickering, a temporal stabilization algorithm is called to denoise the image flickering; thus, a corresponding denoising model specifically for complex noises is constructed, effectively removing vertical bar noise, snowflake noise, and flickering phenomena in the video. Through the combination of deep learning models, the efficiency and quality of video restoration are significantly improved, and the deficiencies in processing complex noises, detail restoration, and high-resolution video restoration in the prior art are solved by an optimized super-resolution algorithm model.

[0054] The deep learning super-resolution model adopted in this application can adaptively adjust the edges and details of video frames, making the enhanced image clearer, especially suitable for the playback requirements of large screens and high-definition displays. After super-resolution enhancement, the detail performance of the video is clearer, especially the edges and high-frequency parts, and the detail restoration ability has been significantly improved, with a more prominent visual effect when displayed on a large screen.

[0055] The present invention significantly improves the efficiency of video processing by optimizing the structure and calculation process of the deep learning model. Specifically, it is manifested as: saving computing resources. The deep learning model of the present invention optimizes the calculation process and reduces the computational overhead when processing high-resolution videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Schematic flowchart of a video restoration and enhancement method for adaptive image processing provided by an embodiment of the present application;

[0057] Figure 2 Structural diagram of a denoising model based on a convolutional neural network provided by an embodiment of the present application;

[0058] Figure 3 Structural diagram of a denoising model based on a generative adversarial network in an embodiment of the present application;

[0059] Figure 4 Structural diagram of a hybrid model based on a convolutional network and an adversarial network provided by an embodiment of the present application;

[0060] Figure 5 Structural diagram of a super-resolution model of deep learning provided by an embodiment of the present application;

[0061] Figure 6 Flowchart of a video restoration method for adaptive image processing provided by an embodiment of the present application;

[0062] Figure 7 Specific process of the temporal stabilization algorithm provided by an embodiment of the present application;

[0063] Figure 8 Detailed flowchart of a video restoration and enhancement method for adaptive image processing provided by an embodiment of the present application;

[0064] Figure 9 Detailed process of denoising processing for vertical bar noise provided by an embodiment of the present application. Specific embodiments

[0065] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In one embodiment, as Figure 1 shown, a video restoration and enhancement method for adaptive image processing is provided, and the method includes:

[0067] Step S1: Construct a denoising model based on a convolutional neural network; the denoising model based on a convolutional neural network is composed of a conv block, a dilated convolutional kernel, a residuai Block, a noise prediction, and a noise removal module, so as to perform noise prediction according to the extracted features, and then output a denoised image by subtracting the predicted noise from the original image.

[0068] Specifically, as Figure 2As shown in the figure, it is the structural diagram of a denoising model based on a convolutional neural network; specifically, it includes multiple conv block blocks: used for extracting shallow features, two dilated convolutional kernels of different sizes, a conv block for feature map splicing, a convolutional kernel for dimensionality reduction, multiple residuai Block (continuous residual blocks), a noise prediction and a noise removal module.

[0069] The convolutional neural network model is constructed by the following equation:

[0070]

[0071] Where, is the pixel value after denoising, W is the convolutional kernel, b is the bias, and σ is the activation function.

[0072] Step S2: Construct a denoising model based on a generative adversarial network; the generator in the denoising model based on a generative adversarial network consists of multiple convolutional layers, multiple residual modules, and 1 spatial channel attention. The residual module contains two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules, and 1 spatial channel attention adopt a skip connection with a symmetric structure.

[0073] In step S2, as Figure 3 shown, the generator of the denoising model based on a generative adversarial network is composed of three convolutional layers, two residual modules, spatial channel attention, and three convolutional layers connected in sequence. At the same time, the network also includes a skip connection with a symmetric structure, making the network have better convergence. The skip connection of the residual module will be input to the spatial channel attention module to further extract important spatial features such as edges and textures in the feature map, and suppress unimportant background regions or noise regions. Each residual block contains two convolutional layers, two batch normalizations, and two ReLU activation functions to increase the depth of the network. The generative adversarial network model combining the above advantages can improve the denoising effect of images and is suitable for denoising tasks with texture features such as snow noise.

[0074] The generator and discriminator are constructed by the following formulas: Generator:

[0075] G(z) = Conv(ReLU(BatchNorm(z)))

[0076] Where, z is the input noise vector, Conv represents the convolutional operation, ReLU is the activation function, and BatchNorm is the batch normalization operation.

[0077] Discriminator:

[0078] D(I) = σ(W d *I + b d)

[0079] Among them, I is the input image, W d is the weight of the discriminator, b d is the bias term, and σ is the activation function.

[0080] Step S3: Construct a hybrid model based on a convolutional network and an adversarial network; the hybrid model based on a convolutional network and an adversarial network is composed of a convolutional module and a generative adversarial network module.

[0081] As Figure 4 shown, constructing the hybrid model based on a convolutional network and an adversarial network is that the convolutional layer, the convolutional module, and the generative adversarial network module are connected in sequence; among them, the convolutional module is composed of 2 convolutions, splitting, Block blocks, transposed convolution blocks, and splicing; the generative adversarial network module is composed of 2 convolutions, a generative adversarial network, and 4 Block blocks. In the denoising task, compared with the ordinary upsampling method (such as interpolation), the transposed convolution can learn a more complex mapping relationship. It helps to restore the detailed information of the image. And the residual block can enhance the expression ability of the model and avoid over-smoothing. Combining the above characteristics, this model can handle various types of noise (such as Gaussian noise, salt-and-pepper noise, real noise, etc.).

[0082] Step S4: Construct a super-resolution model of deep learning; the super-resolution model of deep learning includes a shallow network, a deep network, and adaptive convolution for adaptively restoring the edges and texture details of the image; the super-resolution model of deep learning is optimized using a multiple loss function.

[0083] Specifically, the super-resolution model is mainly composed of a convolutional network and an adversarial network. As Figure 5 shown, it includes the following:

[0084] (1) Shallow network (rough super-resolution processing):

[0085] Function: It is used to first extract basic features from the denoised low-resolution image using a convolutional layer, quickly increase the resolution, then use the activation function to process the non-linear transformation, and finally use the pooling layer to reduce the dimension to obtain a medium-resolution image;

[0086] Implementation process: Use a small convolutional kernel (such as 3×3) for feature extraction:

[0087] F1(x,y) = σ(W1 * I low (x,y) + b1)

[0088] Among them, I low (x,y) is the input low-resolution image, F1(x,y) is the output feature map, W1 is the convolutional kernel, b1 is the bias, and σ is the activation function.

[0089] Dimensionality reduction through a pooling layer (such as max pooling):

[0090] P(x,y) = max(F1(x,y))

[0091] Thereby improving the efficiency of subsequent processing.

[0092] (2) Deep network (detail restoration):

[0093] Function: Restore the edge and texture details of the image through deeper network layers. Specifically, perform multiple feature extractions on the medium-resolution image output by the shallow network using multiple convolutional blocks; among them, the multiple convolutional blocks are connected using residuals; then use the activation function to process the non-linear transformation;

[0094] Implementation process: Introduce residual convolutional blocks (Residual Blocks):

[0095] F2(x,y) = F1(x,y) + R(F1(x,y))

[0096] Among them, R represents the residual module, which can alleviate the problem of gradient disappearance. Use cross-layer skip connections to enhance the transmission of detail features.

[0097] (3) Adaptive convolution (fine-tuning):

[0098] Function: Dynamically adjust the convolutional kernels in the shallow network and the deep network, and perform adaptive detail enhancement on the key regions of the image (such as edges or high-frequency regions).

[0099] Implementation process: Dynamically adjust the size and weight of the convolutional kernel according to the image region.

[0100] W adaptive = f(region, feature map)

[0101] Among them, f is the weight generation function, and region represents the region type (edge or flat region). Generate the fine-tuned image:

[0102] I enhances (x,y) = W adaptive * F2(x,y)

[0103] I enhances (x,y) is the adjusted image result.

[0104] (4) Discriminator: Used to evaluate the authenticity of the image.

[0105] (5) Design of the hybrid loss function:

[0106] To ensure the high quality of the generated images, the present invention designs a hybrid loss function to optimize the model by combining multiple metrics:

[0107] Perceptual Loss

[0108]

[0109] Among them, The high-level feature map of the image in the pre-trained network (such as VGG). It is to ensure that the repaired image is close to the real image in terms of perceptual quality.

[0110] Adversarial Loss

[0111] L GAN = E[logD(I real )] + E[log(1 - D(G(z)))]

[0112] Among them, G is the generator and D is the discriminator. The authenticity of the generated image is improved through the generative adversarial network (GAN).

[0113] Mean squared error loss:

[0114]

[0115] Combining the above three losses, the optimization objective is:

[0116] L total = αL perceptual + βL GAN + γL MSE

[0117] Among them, α, β, and γ are weight hyperparameters used to balance the contributions of different losses.

[0118] Step S5: Receive the target video and identify the noise type for each frame image in the target video; the noise types include vertical bar noise, snowflake noise, image flicker, and other specific noises.

[0119] Among them, the edge detection model and the pattern detection model are used to detect the noise features. The following is the formula of the edge detection model:

[0120]

[0121] Among them, I(x, y) is the brightness value of the pixel point, and respectively represent the gradient changes in the horizontal and vertical directions of the image.

[0122] Step S6: Denoise each frame of the image, specifically including: if there are vertical bar noises in the image, call the denoising model based on convolutional neural network to denoise the vertical bar noises; if there are snowflake noises in the image, call the denoising model based on generative adversarial network to denoise the snowflake noises; if there is image flickering in the image, call the temporal stabilization algorithm to denoise the image flickering; if there are other specific noises in the image, call the hybrid model based on convolutional network and adversarial network to denoise.

[0123] Specifically, the specific process of the temporal stabilization algorithm is as Figure 7 shown.

[0124] For the brightness fluctuation problem in time series, calculate the brightness change of each pixel through the temporal analysis of the front and back frames:

[0125]

[0126] Among them, L t (x, y) is the brightness value of the current frame, and ΔL(x, y, t) is the brightness change rate.

[0127] Based on the smoothing process of the temporal frames, adjust the current frame using the brightness mean or median of the front and back frames:

[0128]

[0129] Among them, W t is the weight, which is used to dynamically adjust the smoothing intensity.

[0130] For the detail area, use median smoothing to reduce mutations:

[0131]

[0132] This method ensures the coherence between video frames while eliminating brightness fluctuations.

[0133] Step S7: Use the super-resolution model of the deep learning to enhance the image quality of the denoised image.

[0134] In step S7, the present invention uses the deep learning super-resolution algorithm to enhance the image quality of the video, which can improve the low-resolution image to high-definition resolution and enhance the detail information in the image at the same time. Through the layer-by-layer processing of the convolutional neural network, the model can adaptively enhance the edges and details of the video, making the video present a clearer and more realistic visual effect on the large screen and high-definition display. This super-resolution technology not only improves the resolution of the video, but also maintains the details and realism of the image, making up for the defect that the traditional super-resolution method cannot restore details.

[0135] In the above video restoration and enhancement method for adaptive image processing, the noise in the image is classified, and the corresponding constructed deep learning noise model is used for removal. For example, for vertical bar noise, a denoising model based on a convolutional neural network is called to denoise the vertical bar noise; in the denoising model based on a convolutional neural network, by introducing shallow feature extraction and multi-scale processing, the model can capture more image details, and the noise distribution is explicitly modeled through a noise prediction module to improve the denoising accuracy.

[0136] A denoising model based on a generative adversarial network is called to denoise snowflake noise; in the denoising model based on a generative adversarial network for snowflake noise, by stacking residual modules, spatial channel attention and other modules in the generator network, each residual block contains two convolutional layers, two batch normalizations, and two ReLU activation functions to increase the depth of the network. A skip connection with a symmetric structure is also included in the network, making the network have better convergence. The skip connection of the residual module is input to the spatial channel attention module to further extract important spatial features such as edges and textures in the feature map and suppress unimportant background regions or noise regions. The generative adversarial network model combining the above advantages can improve the denoising effect of the picture and is suitable for denoising tasks with texture features such as snowflake noise. Other noises are processed by a hybrid model of a convolutional network and an adversarial network, where the transposed convolution in the convolutional module can learn more complex mapping relationships. This helps to restore the detail information of the image. The residual block can enhance the expressive ability of the model and avoid over-smoothing. Combining the above characteristics, this model can process various types of noises (such as Gaussian noise, salt-and-pepper noise, real noise, etc.). For image flickering, a temporal stabilization algorithm is called to denoise the image flickering; thus, a corresponding denoising model for specifically processing complex noises is constructed, effectively removing the vertical bar noise, snowflake noise, and flickering phenomenon in the video. Through the combination of deep learning models, the efficiency and quality of video restoration are significantly improved, and the deficiencies in processing complex noises, detail restoration, and high-resolution video restoration in the prior art are solved by an optimized super-resolution algorithm model.

[0137] The deep learning super-resolution algorithm adopted in this method makes the enhanced image clearer by adaptively adjusting the edges and details of video frames, especially suitable for the playback requirements of large screens and high-definition monitors. After super-resolution enhancement, the details of the video are more clearly presented, especially the edges and high-frequency parts, and the detail restoration ability has been significantly improved, with a more prominent visual effect when displayed on a large screen.

[0138] The present invention significantly improves the efficiency of video processing by optimizing the structure and calculation process of the deep learning model. Specifically, it is manifested as: saving computing resources. The deep learning model of the present invention optimizes the calculation process and reduces the computational overhead when processing high-resolution videos.

[0139] As Figure 8 shown, the detailed process of this application includes the following content:

[0140] (1) Noise classification and repair:

[0141] During the video repair process, it is first necessary to identify and classify various noises in the video. To efficiently process the noise, the present invention uses a noise classifier to identify the noise in the video.

[0142] (2) Noise classification and processing:

[0143] Vertical bar noise: Vertical bar noise mainly appears as obvious vertical stripes in the video frame. This kind of noise is usually related to hardware problems or data transmission and storage during the video shooting process. For this reason, the present invention adopts a de-stripe algorithm, which extracts features in the video frame through a convolutional neural network (CNN) and effectively removes these structural noises. The convolutional layer of the CNN can capture image features at different scales, thus removing the stripes without affecting other image details.

[0144] Snowflake noise: Snowflake noise is a kind of random noise, usually manifested as granular spots in the video picture. To effectively remove snowflake noise, the present invention adopts a denoising model based on a generative adversarial network (GAN). Through the adversarial training of the generator and the discriminator, GAN can not only remove snowflake noise, but also enhance the stability and clarity of the video. The generator is responsible for generating the repaired image, and the discriminator judges whether the image conforms to the characteristics of the real video. The two confront each other, thus realizing high-quality noise removal.

[0145] Image flicker: Flicker usually appears in the video, manifested as sudden changes in brightness. To solve this problem, the present invention adopts a temporal stabilization algorithm to smooth the brightness fluctuations between video image frames, reduce sudden changes in brightness, and thus ensure the smoothness and visual coherence of the video.

[0146] Other specific noises, noises other than vertical bar noise, snowflake noise, and image flicker are other specific noises. The present invention performs denoising processing through a hybrid model based on a convolutional network and an adversarial network.

[0147] (3) Denoising model selection:

[0148] After the noise classification and recognition are completed, the present invention selects the most suitable denoising model according to different types of noise. Through the effective operation of the classifier, we can adopt different denoising techniques for different noises to ensure that each noise is optimally processed.

[0149] The present invention adopts a hybrid deep learning model that combines a convolutional neural network (CNN) with a generative adversarial network (GAN) to achieve efficient noise removal and image restoration. The specific technical solutions are as follows:

[0150] a: Model architecture:

[0151] Convolutional neural network (CNN): As a deep learning framework, CNN is good at extracting multi-level features from images and is especially suitable for image denoising and image restoration tasks. In the present invention, CNN is used to extract the structured noise features in video frames and effectively remove vertical bar noise.

[0152] Generative adversarial network (GAN): GAN adopts an adversarial training mechanism of a generator and a discriminator, enabling the generator to restore image details while removing noise. The discriminator judges the similarity between the generated image and the real image, thereby continuously optimizing the performance of the generator. The GAN model in the present invention is mainly used to remove snowflake noise and enhance the stability of the video.

[0153] b: Training process:

[0154] To ensure the good performance of the model in various video noise processing, the present invention uses a large-scale high-quality video dataset for training. During the training process, the hyperparameters and architecture of the model are continuously optimized, enhancing the generalization ability of the model. By finely adjusting the filters of the convolutional layer and the hierarchical structure of the network, the model can efficiently process different types of noise and ensure that the quality of the restored video is not affected.

[0155] c: Processing and restoration:

[0156] After training, the deep learning model of the present invention can process each frame, remove noise and enhance the details of the video. Through the adjustment of the backpropagation algorithm and the optimization algorithm, the model can accurately repair various defects in the video and ensure that the restored video maintains high quality and visual smoothness.

[0157] (4) Manual assisted processing and face enhancement:

[0158] Although most video restoration work can be completed by automated processing techniques, some complex scenarios still require manual assistance. Based on automated processing, the present invention adds a manual repair link to ensure the repair effect of particularly complex or important picture areas. The specific technical solutions are as follows:

[0159] a: Automated processing:

[0160] For most image defect problems, such as small - scale noise, low - frequency noise, etc., they can be effectively repaired through the automatic processing of the deep - learning model. For large - scale noise, the model performs adaptive repair by extracting the features of the surrounding areas.

[0161] b: Manual repair:

[0162] In some particularly complex scenarios, such as blurred facial images, loss of details in key areas, regional flickering, etc., the automatic repair model may not achieve the ideal visual effect. Therefore, the present invention introduces a manual - assisted repair mechanism to specifically handle image problems that are difficult to automatically repair, ensuring that the quality of the final repaired image reaches the best.

[0163] (5) Repair and enhancement of super - resolution algorithm:

[0164] After noise removal and manual repair are completed, the present invention further uses a deep - learning super - resolution algorithm to enhance the video. Super - resolution technology can upgrade a low - resolution video to a higher resolution and enhance and restore detail information during the enhancement process.

[0165] a: Super - resolution model architecture:

[0166] The present invention adopts a super - resolution algorithm based on a convolutional neural network (CNN). By processing video image frames layer by layer, it achieves the effect of gradually increasing the image resolution. This super - resolution algorithm model adaptively adjusts the edges and details of the low - resolution image, so that the enlarged image still maintains an excellent visual effect on large screens and high - definition displays.

[0167] b: Enhancement effect:

[0168] The super - resolution algorithm not only improves the resolution of the video but also enhances the detail performance of the video, making the finally output video clearer and more realistic, meeting the requirements of high - quality video display.

[0169] (6) Workflow and technological innovation points:

[0170] The workflow of the present invention is shown in the figure:

[0171] Video input: Obtain a low - resolution or noise - rich video to be processed.

[0172] Noise classification and removal: Identify the noise in the video through a classifier and use CNN and GAN models to remove the noise.

[0173] Deep - learning repair: Further repair the details in the video based on a deep - learning model.

[0174] Manual - assisted repair: Manually repair particularly complex scenes.

[0175] Super-resolution enhancement: Use super-resolution algorithms to improve the video resolution and enhance the detail performance.

[0176] Video output: Output the final repaired and enhanced video, meeting the high-definition standard.

[0177] The main technical innovation of this invention lies in the combination of deep learning models, which significantly improves the efficiency and quality of video repair. And through the optimized super-resolution algorithm and artificial-assisted repair mechanism, it solves the deficiencies in dealing with complex noise, detail restoration, and high-resolution video repair in the prior art, providing a more efficient and accurate video repair solution.

[0178] In one embodiment, the method for film and television video repair and super-resolution enhancement based on deep learning proposed by this invention has obvious advantages and remarkable effects compared with the existing video repair technologies and super-resolution technologies, which are specifically manifested in the following aspects:

[0179] (1) Improve video repair accuracy and detail retention:

[0180] Existing video repair methods usually rely on rule-based image processing techniques, such as image filtering and interpolation. Although these methods can remove some noise, they often lead to the loss of video details, especially in dealing with complex noise (such as stripe, flicker, and snow noise). In contrast, this invention adopts a hybrid model that combines convolutional neural network (CNN) and generative adversarial network (GAN) in deep learning, which can accurately restore the lost details while denoising, and retain the naturalness and authenticity of the video. The specific effects are as follows:

[0181] Remarkable noise removal effect: Through the use of targeted denoising models, vertical bar noise, snow noise, and flicker phenomena in the video have been effectively removed.

[0182] Detail retention and restoration: After being processed by the hybrid model combining CNN and GAN, the details (such as human facial features, textures, etc.) in the video image are maintained more naturally and clearly, and the detail restoration accuracy has made great progress compared with the existing methods.

[0183] (2) Significantly improve the video super-resolution effect:

[0184] Traditional super-resolution methods, such as interpolation-based techniques, usually cannot effectively recover the lost details and are prone to causing image blurring. The deep learning super-resolution algorithm adopted by the present invention adaptively adjusts the edges and details of video frames, making the enhanced image clearer, especially suitable for the playback requirements of large screens and high-definition displays. After super-resolution enhancement, the details of the video are more clearly presented, especially the edges and high-frequency parts, and the detail recovery ability has been significantly improved, with a more prominent visual effect when displayed on a large screen.

[0185] (3) Improve processing efficiency and computing performance:

[0186] Compared with existing video restoration and super-resolution methods, the present invention significantly improves the efficiency of video processing by optimizing the structure and computing process of the deep learning model. Specifically, it shows that: in terms of saving computing resources, the deep learning model of the present invention optimizes the computing process and reduces the computing overhead when processing high-resolution videos. Experimental data show that for the same restoration task, the processing speed of the present invention is about 50% higher than that of traditional deep learning methods, significantly reducing the demand for computing resources.

[0187] (4) Reduce the need for manual intervention and simplify the operation process:

[0188] The present invention integrates an automated restoration and enhancement process, greatly reducing the need for manual restoration and improving the simplicity and efficiency of operation. Manual restoration is mainly used to handle particularly complex scenarios or key picture areas, and the automated system can handle most defects, reducing the intensity of manual intervention.

[0189] The innovation of the present invention in video restoration and super-resolution technology not only improves the accuracy of video restoration and detail retention, but also greatly enhances the effect of super-resolution processing, significantly optimizing the computing efficiency, reducing energy consumption and the demand for computing resources. Through the optimization of the deep learning model, the present invention improves the processing speed and automation level while ensuring high-quality restoration, greatly reducing the need for manual intervention and simplifying the operation process. These technical advantages make the present invention have important application value in the field of video restoration and super-resolution enhancement.

[0190] In one embodiment, after calling the denoising model based on a convolutional neural network, the denoising model based on a generative adversarial network, and the hybrid model based on a convolutional network and an adversarial network, it further includes: determining whether the denoised image flickers, and if so, using the temporal front and back smoothing algorithm to repair the flicker. Figure 9As shown, it is the detailed process for processing vertical bar noise, including: vertical bar noise recognition, generating a mask image corresponding to the vertical bar area, using a denoising model based on a convolutional neural network to perform vertical bar filling and repair with the denoising model based on a convolutional neural network, determining whether there is flickering, and if so, using a temporal stabilization algorithm to remove the flicker, and outputting the denoised video frame.

[0191] In one embodiment, a video restoration and enhancement system for adaptive image processing is provided. The system includes:

[0192] A denoising model construction module based on a convolutional neural network: used to construct a denoising model based on a convolutional neural network; the denoising model based on a convolutional neural network is composed of a conv block, a dilated convolutional kernel, a residuai Block, a noise prediction, and a noise removal module, to perform noise prediction according to the extracted features, and then output a denoised image by subtracting the predicted noise from the original image;

[0193] A denoising model construction module based on a generative adversarial network: used to construct a denoising model based on a generative adversarial network; the generator in the denoising model based on a generative adversarial network consists of multiple convolutional layers, multiple residual modules, and 1 spatial channel attention. The residual module contains two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules, and 1 spatial channel attention adopt a symmetric structure of skip connections;

[0194] A hybrid model construction module based on a convolutional network and an adversarial network: used to construct a hybrid model based on a convolutional network and an adversarial network; the hybrid model based on a convolutional network and an adversarial network includes a convolutional module and a generative adversarial network module;

[0195] A super-resolution model construction module for deep learning: used to construct a super-resolution model for deep learning; the super-resolution model for deep learning includes a shallow network, a deep network, and adaptive convolution, to adaptively restore the edges and texture details of the image; the super-resolution model for deep learning is optimized using a multiple loss function;

[0196] A noise recognition module: used to receive a target video and identify the noise type for each frame image in the target video; the noise types include vertical bar noise, snowflake noise, image flickering, and other specific noises;

[0197] Noise removal module: used to remove noise from each frame of the image, specifically including: if there is vertical bar noise in the image, call the denoising model based on convolutional neural network to denoise the vertical bar noise; if there is snowflake noise in the image, call the denoising model based on generative adversarial network to denoise the snowflake noise; if there is image flickering in the image, call the temporal stabilization algorithm to denoise the image flickering; if there is other specific noise in the image, call the hybrid model based on convolutional network and adversarial network to denoise;

[0198] Image quality enhancement module: used to enhance the image quality of the denoised image using the super-resolution model of the deep learning.

[0199] For the specific limitations of a video restoration and enhancement system for adaptive image processing, reference can be made to the limitations of a method for video restoration and enhancement of adaptive image processing in the above text, which will not be elaborated here. Each module in the above video restoration and enhancement system for adaptive image processing can be implemented in whole or in part by software, hardware and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above modules.

[0200] In one embodiment, a computer device is provided, and this computer device can be a server. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes the above method for video restoration and enhancement of adaptive image processing.

[0201] In one embodiment, a computer program product is further provided, including a computer program / instructions, and when the computer program / instructions are executed by the processor, they involve all or part of the processes in the above method of the embodiment.

[0202] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0203] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0204] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A video restoration and enhancement method for adaptive image processing, characterized in that: The method comprises: Constructing a denoising model based on a convolutional neural network; the denoising model based on a convolutional neural network consists of a conv block, a hole convolution kernel, a residuai Block, a noise prediction and a noise removal module, so as to perform noise prediction according to the extracted features, and then output a denoised image according to the original image minus the predicted noise; Constructing a denoising model based on a generative adversarial network; the generator in the denoising model based on a generative adversarial network consists of multiple convolutional layers, multiple residual modules and one spatial channel attention, the residual module includes two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules and one spatial channel attention adopt a symmetrically structured jump connection; Constructing a hybrid model based on a convolutional network and an adversarial network; the hybrid model based on a convolutional network and an adversarial network includes a convolution module and a generative adversarial network module; Constructing a deep learning super-resolution model; the deep learning super-resolution model includes a shallow network, a deep network and an adaptive convolution, so as to adaptively restore the edge and texture details of the image; the deep learning super-resolution model is optimized using multiple loss functions; Receive a target video, and identify the noise type of each frame of the target video; the noise type includes vertical bar noise, snowflake noise, image flicker, and other specific noise; De-noising each frame of the image, specifically including: if there is vertical bar noise in the image, the denoising model based on the convolutional neural network is called to denoise the vertical bar noise; if there is snowflake noise in the image, the denoising model based on the generative adversarial network is called to denoise the snowflake noise; if there is image flicker in the image, the temporal stabilization algorithm is called to denoise the image flicker; if there is other specific noise in the image, the hybrid model based on the convolutional network and the adversarial network is called to denoise; The denoised image is subjected to image quality enhancement using the deep learning super-resolution model.

2. The video restoration and enhancement method of adaptive image processing according to claim 1, characterized in that: The calling of the denoising model based on a convolutional neural network, the denoising model based on a generative adversarial network, and the calling of the hybrid model based on a convolutional network and an adversarial network also includes: determining whether the denoised image flickers, and if so, using the time series forward and backward smoothing algorithm to repair the flicker.

3. The video restoration and enhancement method of adaptive image processing according to claim 1, characterized in that: The timing stabilization algorithm process includes: analyzing the brightness fluctuations of the previous and next frames through timing, calculating the brightness changes of each pixel to identify image flicker; using image segmentation technology to divide the detail area or background area; for the image flicker in the detail area, dynamically adjusting the weight according to the timing changes, and eliminating the fluctuations through weighted median value smoothing; for the image flicker in the background area, the median is used to eliminate the fluctuations.

4. The video restoration and enhancement method of adaptive image processing according to claim 3, characterized in that: The calculation of the brightness change of each pixel is achieved by the following formula: Among them, L t (x, y) is the brightness value of the current frame, and ΔL(x, y, t) is the brightness change.

5. The video restoration and enhancement method of adaptive image processing according to claim 3, characterized in that: The elimination of fluctuations is performed by the following formula: Among them, W t is the weight.

6. The video restoration and enhancement method of adaptive image processing according to claim 1, characterized in that: The deep learning super-resolution model is specifically implemented by the following formula: The shallow network is implemented by the following formula: Use small convolution kernels for feature extraction; F1(x,y)=σ(W1*I low (x,y)+b1) Among them, I low (x, y) is the input low-resolution image, F1(x, y) is the output feature map, W1 is the convolution kernel, b1 is the bias, and σ is the activation function; Dimensionality reduction is achieved through the pooling layer, and the maximum pooling layer is implemented by the following formula: P(x,y)=max(F1(x,y)) Among them, F1(x,y) is the value of the input feature map at position (x,y), P(x,y) is the value of the output feature map at position (x,y), and the max(·) operation represents the maximum value of all values ​​in the pooling window; The deep network introduces the residual convolution block: F2(x,y)=F1(x,y)+R(F1(x,y)) Among them, R represents the residual module, which can alleviate the gradient vanishing problem and enhance the transmission of detail features by using cross-layer skip connections; The adaptive convolution is constructed by the following formula: W adaptive =f(region,feature map) Among them, the weight generation function W is a function that dynamically generates the weight of the convolution kernel, f is a learnable function used to dynamically generate the weight of the convolution kernel according to the region type and feature map, region represents the region type processed by the current convolution operation, and feature map represents the input feature map; Generate a fine-tuned image: I enhances (x,y)=W adaptive *F2(x,y) I enhances (x,y) is the resized image result.

7. The video restoration and enhancement method of adaptive image processing according to claim 1, characterized in that: The optimization using multiple loss functions includes: The final loss function is obtained by combining multiple loss functions through the following formula: L total =αL perceptual +βL GAN +γL MSE Among them, α, β, γ are weight hyperparameters, L perceptual is the perceptual loss function, L GAN is the adversarial loss function, L MSE is the mean square error loss function.

8. A video restoration and enhancement system for adaptive image processing, characterized in that: The system comprises: Denoising model building module based on convolutional neural network: used to build a denoising model based on convolutional neural network; the denoising model based on convolutional neural network consists of conv block, hole convolution kernel, residuai Block, noise prediction and noise removal modules, so as to predict noise according to the extracted features, and then output a denoised image according to the original image minus the predicted noise; A denoising model construction module based on a generative adversarial network: used to construct a denoising model based on a generative adversarial network; the generator in the denoising model based on a generative adversarial network is composed of multiple convolutional layers, multiple residual modules and one spatial channel attention, and the residual module includes two convolutional layers, two batch normalizations, and two ReLU activation functions; the multiple convolutional layers, multiple residual modules and one spatial channel attention adopt a symmetrically structured jump connection; A module for constructing a hybrid model based on a convolutional network and an adversarial network: used to construct a hybrid model based on a convolutional network and an adversarial network; the hybrid model based on a convolutional network and an adversarial network includes a convolution module and a generative adversarial network module; A deep learning super-resolution model construction module: used to construct a deep learning super-resolution model; the deep learning super-resolution model includes a shallow network, a deep network, and an adaptive convolution, so as to adaptively restore the edge and texture details of the image; the deep learning super-resolution model is optimized using multiple loss functions; Noise recognition module: used to receive the target video and identify the noise type of each frame in the target video; the noise types include vertical bar noise, snowflake noise, image flicker, and other specific noises; Noise removal module: used to remove noise from each frame of the image, specifically including: if there is vertical bar noise in the image, the denoising model based on the convolutional neural network is called to denoise the vertical bar noise; if there is snowflake noise in the image, the denoising model based on the generative adversarial network is called to denoise the snowflake noise; if there is image flicker in the image, the temporal stabilization algorithm is called to denoise the image flicker; if there is other specific noise in the image, the hybrid model based on the convolutional network and the adversarial network is called to denoise; Image quality enhancement module: used to enhance the image quality of the denoised image using the deep learning super-resolution model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images