Video processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and particularly provides a video processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining an initial video which comprises at least two first video frames; for at least two first video frames, the first video frames are input into a first neural network model to obtain a first feature map, and the first feature map comprises high-frequency information and dark channel information; inputting the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; inputting the second feature map and the first video frame into a third neural network model for defuzzification processing to obtain a second video frame; and combining the at least two second video frames to obtain a target video. According to the embodiment of the invention, the fuzzy region in the continuous frames is processed while the super-resolution is ensured, and the quality of the shot video is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video processing method, apparatus, device and storage medium. Background Art

[0002] Image super resolution (SR) reconstruction technology is to restore one or more low-resolution (LR) images to corresponding high-resolution (HR) images. Simply put, it uses the feature prior information in the video image to simulate the image degradation process, so as to obtain a more detailed video.

[0003] There are various sports scenes in life, such as fast-running athletes, high-speed trains, constantly moving dragonflies, etc. When using a camera to shoot sports scenes, due to the rapid movement of objects, the captured images or videos may have blurred areas such as ghosting.

[0004] Although related technologies can clarify blurred images, how to achieve super-resolution reconstruction while achieving clarity is an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a video processing method, device, equipment and storage medium, which can process blurred areas in continuous frames while ensuring super-resolution, thereby improving the quality of captured videos.

[0006] In a first aspect, the present application provides a video processing method, the method comprising: obtaining an initial video, the initial video comprising at least two first video frames; for the at least two first video frames, inputting the first video frame into a first neural network model to obtain a first feature map, wherein the first feature map comprises high-frequency information and dark channel information; inputting the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; inputting the second feature map and the first video frame into a third neural network model for deblurring processing to obtain a second video frame; and merging at least two second video frames to obtain a target video.

[0007] In a second aspect, the present application provides a video processing device, which includes: an initial video acquisition module, used to acquire an initial video, the initial video including at least two first video frames; a first feature map determination module, used to input the first video frame into a first neural network model for at least two first video frames to obtain a first feature map, wherein the first feature map includes high-frequency information and dark channel information; a second feature map determination module, used to input the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; a second video frame determination module, used to input the second feature map and the first video frame into a third neural network model for deblurring processing to obtain a second video frame; and a target video determination module, used to merge at least two second video frames to obtain a target video.

[0008] In a third aspect, the present application provides a video processing device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the video processing method as described in the first aspect above.

[0009] In a fourth aspect, the present application provides a storage medium, which may be a computer-readable storage medium, storing a computer program thereon, which, when executed by a processor, implements the video processing method in the first aspect described above.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the video processing method as described in any one of the first aspects above is implemented.

[0011] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0012] The embodiments of the present application provide a video processing method, apparatus, device and storage medium, the method comprising: obtaining an initial video, the initial video comprising at least two first video frames; for the at least two first video frames, inputting the first video frame into a first neural network model to obtain a first feature map, wherein the first feature map comprises high-frequency information and dark channel information; inputting the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; inputting the second feature map and the first video frame into a third neural network model for deblurring processing to obtain a second video frame; and merging at least two second video frames to obtain a target video.

[0013] Since the first neural network model is used to extract high-frequency information and dark channel information in the first video frame as auxiliary information to clarify the first video frame, the deblurring processing capability of the model is enhanced. After the second neural network model is used to perform super-resolution reconstruction, the third neural network model is used to perform deblurring processing. While ensuring super-resolution, the blurred areas in continuous frames are processed to improve the quality of the captured video. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 A schematic diagram of a video processing method according to an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the structure of a video processing model provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of a generator network provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of the structure of the discriminator network provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of a flow chart of a method for determining a second characteristic map provided in an embodiment of the present application;

[0021] Figure 6 A comparison chart of super-resolution reconstruction effects provided in the embodiments of the present application;

[0022] Figure 7 A comparison chart of the deblurring effects provided in the embodiments of the present application;

[0023] Figure 8 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] SR reconstruction technology is to restore one or more LR images to the corresponding HR images. In simple terms, it is to use the feature prior information in the video image to simulate the image degradation process, so as to obtain a more detailed video.

[0031] With the popularity of ultra high definition (UHD) display devices, people's demand for high resolution of images / videos in life scenes has become more and more urgent. For example, in the fields of film and television entertainment, medical images, and biometrics. In the field of film and television, super-resolution technology can restore old film and television works through intelligent coloring, video enhancement, deblurring and other technologies. In the medical field, super-resolution technology can improve the clarity of medical images and obtain more accurate lesion information. In the field of biometrics, super-resolution technology can significantly enhance detail textures and edge contours and improve the accuracy of target detection. In summary, super-resolution technology has fully demonstrated its criticality and superiority in many fields such as cultural life and industrial production.

[0032] Among them, SR is a kind of image processing technology that aims to convert low-resolution images into high-resolution images. By utilizing the spatial and statistical information in the image, it attempts to infer the missing pixel details, thereby improving the clarity and details of the image.

[0033] Video SR (VSR) algorithms are divided into aligned VSR algorithms and non-aligned VSR algorithms.

[0034] Among them, the aligned VSR algorithm uses optical flow to estimate the motion information between frames, and then performs motion compensation on the obtained motion information between frames.

[0035] The non-aligned VSR algorithm does not perform alignment before the super-resolution reconstruction process. The non-aligned VSR algorithm only uses the spatial relationship between frames. Considering the temporal relationship between frames, the reconstruction results are often not ideal. The motion estimation and compensation of the aligned VSR algorithm improves the performance of VSR to a certain extent. However, when there is strong light or large motion in the video, the accuracy of the optical flow estimated by this method is difficult to guarantee.

[0036] The non-aligned VSR algorithm only uses the spatial relationship between frames. Considering the temporal relationship between frames, the reconstruction results are often not ideal. When processing videos, the image super-resolution method regards the video as multiple independent images and reconstructs the video sequence as a single frame. It ignores the temporal sequence information of the video and lacks further mining of video features. It is necessary to use inter-frame information to obtain extended information.

[0037] In existing video super-resolution methods, approximate calculation methods are used to estimate the impact of time series on the information between video frames. The most commonly used interpolation methods include bicubic interpolation, bilinear interpolation, and nearest neighbor interpolation. However, the results are not satisfactory, and some images are over-smoothed. In addition, these methods are computationally efficient and suitable for basic parsing tasks and situations with limited computing resources. The above-mentioned video super-resolution algorithms can only perform super-resolution reconstruction on low-resolution videos, and cannot achieve video deblurring.

[0038] There are various sports scenes in life, such as fast-running athletes, high-speed trains, constantly moving dragonflies, etc. When using a camera to shoot sports scenes, due to the rapid movement of objects, the captured images or videos may have blurred areas such as ghosting.

[0039] Although related technologies can clarify blurred images, how to achieve super-resolution reconstruction while achieving clarity is an urgent problem to be solved.

[0040] To solve one or more of the above-mentioned technical problems, an embodiment of the present application provides a video processing method, which includes: obtaining an initial video, the initial video includes at least two first video frames; for the at least two first video frames, inputting the first video frame into a first neural network model to obtain a first feature map, wherein the first feature map includes high-frequency information and dark channel information; inputting the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; inputting the second feature map and the first video frame into a third neural network model for deblurring processing to obtain a second video frame; and merging at least two second video frames to obtain a target video.

[0041] Since the first neural network model is used to extract high-frequency information and dark channel information in the first video frame as auxiliary information to clarify the first video frame, the deblurring processing capability of the model is enhanced. After the second neural network model is used to perform super-resolution reconstruction, the third neural network model is used to perform deblurring processing. While ensuring super-resolution, the blurred areas in continuous frames are processed to improve the quality of the captured video.

[0042] The video processing method provided by the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0043] Figure 1 This is a flowchart of a video processing method in an embodiment of the present application. This embodiment can be applied to super-resolution reconstruction and deblurring of videos. The method can be executed by a video processing device. The video processing device can be implemented in software and / or hardware. The video processing device can be configured in an electronic device.

[0044] like Figure 1 As shown, the video processing method provided in the embodiment of the present application mainly includes steps S101-S105.

[0045] S101. Acquire an initial video, where the initial video includes at least two first video frames.

[0046] Among them, video generally refers to a media form that simulates dynamic effects by continuously playing a sequence of images. Each video frame is a separate picture in a video sequence. When these images are displayed continuously at a certain rate (frame rate, usually expressed in frames per second or fps), the human eye will perceive motion. The initial video can be understood as a video captured by image acquisition without past blurring and super-resolution reconstruction, wherein the first video frame refers to any video frame included in the initial video. Furthermore, the initial video can be understood as a blurred and low-resolution video. The first video frame can be understood as any blurred and low-resolution video frame in the initial video.

[0047] Acquiring the initial video includes: acquiring a video stored in the electronic device as the initial video, or downloading a video from the Internet as the initial video. For example, acquiring a video stored in the "Album" of the smart terminal as the initial video. Or, downloading a film or TV series clip from the Internet as the initial video.

[0048] Acquiring the initial video includes: using the camera video stream captured by the camera of the smart terminal as the initial video. Further, the camera captures a moving object, such as a high-speed vehicle, a fast-running person, a constantly moving dragonfly, and the like.

[0049] Furthermore, the video processing device is connected to a camera, and the camera transmits the collected camera video stream to the video processing device in real time. In other words, the video processing device obtains the camera video stream data collected by the camera in real time.

[0050] Wherein, a video processing model is deployed in the video processing device, and the first video frame is processed by the video processing model to obtain a second video frame, wherein, Figure 2 As shown, the video processing model includes a first neural network model 21, a second neural network model 22 and a third neural network model 23. Among them, the first neural network model includes an edge information extraction network model, the second neural network model includes an adversarial generation network model, and the third neural network model includes a pyramid network model.

[0051] After the video processing model receives the blurred low-resolution first video frame, the first video frame is input into the first neural network model, the first neural network model performs feature extraction on the first video frame to obtain high-frequency information and dark channel information included in the first video frame, the high-frequency information and dark channel information are input into the second neural network model, the second neural network model performs super-resolution reconstruction based on the high-frequency information and dark channel information to obtain a reconstructed second feature map, the reconstructed second feature map and the blurred low-resolution first video frame are input into the third neural network model, the third neural network model is used to divide the multi-scale information into multiple different semantic information, the multi-scale information generates feature maps of corresponding sizes, and the multiple feature maps are merged into a clear super-resolution image.

[0052] The video processing flow is explained below in combination with the structures of various neural network models.

[0053] S102. For at least two first video frames, input the first video frames into a first neural network model to obtain a first feature map, wherein the first feature map includes high-frequency information and dark channel information.

[0054] The first feature map can be understood as a feature map output by the first neural network including high-frequency information and dark channel information. The first neural network model can be understood as an edge information extraction network model, which is mainly used to extract high-frequency information and dark channel information in the first video frame.

[0055] like Figure 2 As shown, the first neural network model includes a high-frequency information extraction network and a dark channel information extraction network. The first video frame is input into the first neural network model to obtain a first feature map, including: inputting the first video frame into the high-frequency information extraction network and the dark channel information extraction network respectively, using the high-frequency information extraction network to extract the high-frequency information in the first video frame, using the dark channel extraction network to extract the dark channel information in the first video frame, and merging the above high-frequency information and dark channel information to obtain the first feature map.

[0056] Among them, high-frequency information usually refers to the parts of the image that change rapidly, which contain subtle details, textures, and edge information of the image. For example, in an image including a building, high-frequency information can refer to the information of the edge of the building.

[0057] The clear areas in natural images usually contain clear edges, while the blurred areas contain blurred edges. The feature map extracted from the first video frame using the high-frequency information extraction network can be used as effective supervision information to enhance the image deblurring ability. At the same time, the transition of high-frequency information can also be carried out smoothly. The high-frequency information in the first video frame is separated in the high-frequency information extraction network for better injection into the GAN.

[0058] The process of extracting high-frequency information using a high-frequency information extraction network mainly includes: using the image degradation model to estimate the blur kernel, and using the estimated blur kernel to perform a deconvolution operation to try to restore the clarity of the image. Apply sharpening filters (such as Laplacian operator, Unsharp Masking, etc.) to enhance the edges and details of the image. Use a high-pass filter to extract high-frequency information from the image. Analyze the image in the rate domain to suppress or enhance specific frequency components. Use detail enhancement algorithms (such as detail magnification, frequency domain filtering, etc.) to further improve the high-frequency information of the image.

[0059] The dark channel information can be understood as the presence of some pixels in the image that are very dark in at least one color channel, and these pixels usually correspond to shadows or areas with low reflectivity in the scene.

[0060] The dark channel information of pixel x in the first video frame I can be expressed by formula (1):

[0061]

[0062] Where x and y represent the positions of pixels, N(x) represents the image patch centered on x, and I c represents the c-th color channel of the first video frame I, r, g, b represent the three color channels of red, green and blue.

[0063] Furthermore, the above formula (1) can be understood as the dark channel information D of the first video frame obtained by calculating the minimum value of each pixel in the three color channels RGB: c (x).

[0064] From formula (1), we can see that the dark channel information describes the minimum intensity value in the image block. In order to better extract the dark pixel information of the image in a dark environment, the first video frame will be subjected to convolution operation and signal element superposition operation.

[0065] The process of extracting dark channel information using the dark channel information extraction network mainly includes: determining the blur kernel; controlling the blur kernel to move in the first video frame according to the set rules; for the image block corresponding to the blur kernel in the first video frame, spatially rearranging according to the intensity value of each pixel in the image block; for each pixel in the image block, multiplying the intensity value of the rearranged pixel with the intensity value corresponding to each pixel in the image block row by row, and then adding all the products to obtain the dark channel information of the pixel.

[0066] Specifically, in order to better extract the dark pixel information of the image in a dark environment, the first video frame is subjected to convolution operation and signal element superposition operation. Specifically, firstly, according to the image degradation model and the image convolution operation under the discrete signal, the intensity value signal of the first video frame is converted and moved according to certain rules; then, the changed signal sequence is element-wise multiplied and added, as shown in formula (2).

[0067]

[0068] Among them, B(x) represents the dark channel information of pixel x, k(i) represents the value of the i-th pixel in blur kernel k, and Ω k represents the support region of the blur kernel. represents the rounding operation of blur kernel k of size S.

[0069] According to the image degradation model, for the dark pixel information of the image, there is more information in the blurred vision than in a given clear image. By performing a convolution operation on the signal, the first video frame can be deblurred on the one hand, and the pixel value of the dark pixel can be increased on the other hand. Due to the sparsity of dark pixels in the first video frame, during the deblurring process, the dark pixel at the (x, y) position will be increased by the weighted average of the surrounding non-dark pixels. By utilizing the sparsity of dark pixels in the first video frame, the blurred image and the clear image can be better distinguished.

[0070] In one possible implementation, dark channel information in a given blurred vision is extracted and used together with the extracted high-frequency information as input to a second neural network model.

[0071] S103: Input the first feature map into a second neural network model for super-resolution reconstruction to obtain a second feature map.

[0072] Among them, the second neural network model includes an adversarial generation network model. In other words, the second neural network model includes a generator network and a discriminator network.

[0073] Among them, the generator network and the discriminator network are the two core components in the generative adversarial network model (GAN). The purpose of the generator network is to generate data samples that are as realistic as possible starting from a random noise vector. These data samples can be images, audio, text, or other types of data. The task of the discriminator network is to distinguish between the fake data samples generated by the generator and the real samples in the real data set.

[0074] In the embodiments of the present application, the structure of the generator network is first briefly introduced.

[0075] like Figure 3 As shown, the generator network includes an input layer 31, a feature extraction layer 32, a residual network 33, a feature fusion layer 34, an upsampling layer 35 and a reconstruction layer 36.

[0076] Furthermore, the generator network is used to perform super-resolution reconstruction on the first feature map to obtain a third feature map, including: using a feature extraction layer to extract shallow features of the first feature map to obtain shallow features; using a residual network to extract deep features of the first feature map to obtain deep features; fusing the shallow features and the deep features to obtain merged features; and reconstructing the merged features to obtain a third feature map.

[0077] Among them, shallow features usually refer to preliminary or low-level features extracted from the first feature map. Deep features refer to higher-level and more abstract information extracted from the first feature map, which can represent more complex visual patterns and object-level semantics.

[0078] The shallow feature extraction layer consists of a convolution layer with a convolution kernel of 9×9, which is mainly responsible for extracting shallow features of low-resolution images.

[0079] In a possible implementation, the residual network includes a first subnetwork and a second subnetwork; the first subnetwork includes a plurality of residual block networks connected in series, the residual block network includes two residual blocks connected in series, the residual block includes a plurality of convolutional layers and a parametric polymorphic linear rectification function PReLU; the second subnetwork includes a plurality of residual dense networks connected in series, the residual dense network includes N convolutional layers, the convolution kernels of the first N-1 convolutional layers are the same, and the convolution kernel of the last convolutional layer is different from the convolution kernels of the first N-1 convolutional layers.

[0080] The residual network can be seen as a combination of improved residual blocks and residual dense blocks. Specifically, the first sub-network includes multiple residual block networks connected in series. The residual block network includes two residual blocks connected in series. Each residual block includes a convolution layer of two convolution kernels 3×3 and skip connection, and a PRELU activation function. The residual block network is used for nonlinear transformation between two convolution layers.

[0081] The second sub-network can be regarded as a residual dense network, which consists of residual dense blocks. Each residual dense block consists of four convolutions. Among them, the convolution kernels of the first three convolution layers are 3×3, which are used to extract deep features of low-resolution images. The last convolution layer uses a 1×1 convolution kernel.

[0082] After merging the first sub-network and the second sub-network in the merging channel, a convolution layer is used to perform convolution calculation on the merged features to achieve the fusion of layout features at different levels.

[0083] In order to improve the utilization rate of model extraction features, the output features of each structural block are merged in RDBG. Feature fusion consists of 1×1 convolution, which can not only greatly increase the nonlinear features while keeping the scale of the feature map unchanged, but also complete the cross-channel information interaction.

[0084] In the embodiment of the present application, the shallow features and the deep features are fused to obtain a merged feature, and the third feature map is reconstructed based on the merged feature, so that the third feature map includes information at different levels.

[0085] Then, the image is upgraded to the target size through an upsampling layer consisting of sub-pixel convolutions. Finally, the corresponding super-resolution image is obtained through a reconstruction layer with a convolution kernel of 9×9. It should be emphasized that the activation function of the reconstruction layer adopts the hyperbolic tangent activation function, as shown in formula (3).

[0086]

[0087] Where tanh(x) represents the hyperbolic tangent activation function, and x represents the input signal of the reconstruction layer.

[0088] In the embodiment of the present application, local residual learning and local features are fused. Local residual learning is implemented using element-wise summation and skip connections in each residual block. In each residual density block, there is not only a local residual learning mechanism but also a feature fusion mechanism. Here, 1×1 convolution is used to adaptively fuse the output features of the previous convolution channel merge to reduce the dimension of the module output features. This structure can not only greatly improve the flow of feature information, but also greatly reduce the parameters of the computational model.

[0089] The loss function of the generator network is explained below.

[0090] The loss function of the generator network can be divided into five parts: L 1 Loss function, L 2 Loss function, perceptual loss function, total variation loss function, and adversarial loss function.

[0091] The generator network loss function is shown in formula (4).

[0092] L G =λ 1 L 1 +λ 2 L 2 +λ 3 L percep +λ 4 L IV +λ 5 L adv (4)

[0093] where λ 1 , 2 , 3 , 4 , 5 It is a conventional parameter obtained through multiple experiments, which is used to adjust the weight ratio of each loss item and optimize the performance of the model. 1 Indicates L 1 Loss function, L 2 Indicates L 2 Loss function, L percep represents the perceptual loss function, L IV represents the total variation loss function, L adv represents the adversarial loss function.

[0094] L 1 Loss function:

[0095] In most cases, L 1 The PSNR value of the reconstructed image of the loss function is relatively low, which can ensure that the similarity of the reconstructed image is high. 1 The mathematical expression of the loss function is shown in formula (5).

[0096]

[0097] Wherein, r represents the upsampling factor, and optionally, r=2 or 4 or 8. HR is the real image, I SR is the reconstructed image, that is, I SR =G(I HR ), W, H represent the real image I HR The width and height of the pixel, w and h represent the horizontal and vertical coordinates of the pixel. represents the pixel value at coordinate (w, h) in the real image, represents the pixel value at coordinate (w, h) in the reconstructed image, ||·|| 1 Indicates L 1 Norm.

[0098] L 2 Loss function:

[0099] In order to obtain a higher PSNR quantization value, in the loss function design of GAN, L is usually added to the generator loss function. 2 As content loss. 2 The mathematical expression of the loss function is shown in formula (6).

[0100]

[0101] Where r represents the upsampling factor, where r = 2 or 4 or 8. W and H represent the real image I HR The width and height of the pixel, w and h represent the horizontal and vertical coordinates of the pixel. represents the pixel value at the coordinate (w, h) in the real image, Represents the pixel value at the coordinate (w, h) in the reconstructed image, that is, I SR =G(I HR ),||·|| 2 Indicates L 2 Norm.

[0102] Perceptual loss function L percep :

[0103] Although L 1 Loss Function and L 2 The combination of loss functions can significantly improve the quality of the reconstructed image, but the reconstructed image is too smooth and cannot overcome the problem of missing high-frequency details. In the embodiment of the present application, perceptual loss is introduced to effectively alleviate the problem of missing high-frequency details. The perceptual loss function can significantly improve the perceptual quality of the reconstructed image and make the visual effect of the image more in line with human aesthetics. Perceptual loss function L percep As shown in formula (7).

[0104]

[0105] Among them, L percep represents the perceptual loss function, I HR represents the real image, I SR represents the reconstructed image, W and H represent the feature dimensions of the specified layer in the generator network, w and h represent the horizontal and vertical coordinates of the pixel points, (I HR ) w,h represents the pixel value at coordinate (w, h) in the real image, represents the pixel value at coordinate (w, h) in the reconstructed image, represents the perceptual function of the real image at coordinate (w, h), Represents the perceptual function of the reconstructed image at coordinate (w, h).

[0106] Furthermore, the generator network uses the pre-trained VGG19 model. Correspondingly, the perceptual loss function L percep As shown in formula (8).

[0107]

[0108] Among them, W 5,4 and H 5,4 Respectively represent the dimensions (width and height) of the feature maps extracted by the 4th and 5th layers of the pre-trained VGG19 model.

[0109] Total variation loss function L IV :

[0110] In order to eliminate noise and artifacts in the reconstructed super-resolution images, the total variation loss is added as a regularization term in the design of the generator loss function.

[0111] The total variation loss function can be expressed as shown in formula (9):

[0112]

[0113] Where x and y represent the reconstructed image I SR The horizontal and vertical gradient functions, I HR is the real image, I SR is the reconstructed image, that is, I SR =G(I HR ), W, H represent the first video frame I HR The width and height of ||·|| 2 Indicates L 2 Norm.

[0114] Adversarial loss function L adv:

[0115] In VSRGAN, there is a generator and two discriminators. That is, the adversarial loss function of the generator contains not only the discriminator D 1 The adversarial loss also includes D 2 To this end, the adversarial loss function is expressed as shown in formula (10):

[0116]

[0117] Among them, I LR Represents the low-resolution image input to the generator, which obeys the corresponding data distribution I LR ∝P LR ,G(I LR ) represents the HR image generated by the generator, which obeys the distribution G(I LR )∝P G ,N represents the number of discriminators in the model, which is defined as N=2.

[0118] The discriminator network is described below.

[0119] The discriminator network includes multiple discriminators with the same structure, and the discriminator is composed of multiple input convolutional layers, global average pooling layers and output convolutional layers in series; among them, the input convolutional layer includes a convolutional layer, an added Leaky layer and a PReLU activation function, and the number of filters in multiple input convolutional layers increases exponentially.

[0120] In the embodiment of the present application, the discriminator network includes two discriminators with the same structure. Specifically, the structure of the above discriminator is improved on the basis of SRGAN. Figure 4 As shown in the figure, among all the convolutional layers of the discriminator, the input convolutional layer can be understood as the convolutional layer before the global average pooling layer, and the output convolutional layer can be understood as the convolutional layer after the global average pooling layer. Furthermore, the convolutional kernels of each input convolutional layer are the same, and the convolutional kernels of each output convolutional layer are the same. The convolutional kernel of the input convolutional layer is different from the convolutional kernel of the output convolutional layer. For example, the convolutional kernel of the input convolutional layer is 3×3, and the convolutional kernel of the output convolutional layer is 1×1.

[0121] Furthermore, the discriminator includes 8 input convolutional layers, each of which consists of a Conv block, a Leaky layer, and a PReLU activation function. It should be noted that the number of filters in the 8 Conv blocks increases exponentially. For example, they are set to 64, 128, 256, 512, 1024, 2048, 4096, 8192, and 16384 in sequence. The step size of the Conv block is alternately set to 1 and 2.

[0122] Furthermore, the discriminator network in the embodiment of the present application does not include a BN layer. Since the BN layer will cause the same batch of images to become mutually dependent, the BN layer is excluded from the discriminator network, which reduces the amount of calculation to a certain extent and can speed up time-consuming training.

[0123] In addition, the discriminator in the embodiment of the present application introduces a global average pooling layer to replace the fully connected layer in the SRGAN network, which can not only effectively prevent the model from overfitting, but also better adapt to the working mechanism of convolution.

[0124] The fully connected layer in the original SRGAN network is replaced with the output convolution layer in the embodiment of the present application. Since the convolution operation is more flexible for the input image, fewer calculation parameters are required.

[0125] Since the baseline loss function of the original SRGAN network is based on the Wasserstein loss of WGAN, the Sigmoid activation function of the output layer is removed. On the one hand, the network is simplified and time is reduced, and redundant components in the discriminator are removed. On the other hand, since there are two completely identical discriminators in the VSRGAN model. In short, the discriminator of the VSRGAN model is an improved form that can fully meet the needs of the model, which improves the robustness and reconstructability of the discriminator network.

[0126] Among them, the discriminator loss function can be divided into two parts, namely and

[0127] In order to further improve the perceptual quality and texture details of the reconstructed image, the model is trained using continuous adversarial between the generator and the two discriminators. The total loss function of the discriminator of the model is shown in formula (11).

[0128] L D =L D1 +L D2 (11)

[0129] It is not difficult to find that when only one set of adversarial games is established between the generator and the discriminator, the reconstructed SR may lose some high-frequency details. Therefore, two discriminator networks are introduced, and there are two sets of adversarial games between the generator and the two discriminators. In this way, the excellent performance of D2GAN can be used to improve the reconstruction effect and obtain more high-frequency details and texture edges.

[0130] Therefore, L of the two discriminators Di The loss function is shown in formula (12):

[0131]

[0132] Among them, i represents two discriminators, IHR is a real image, obeying the corresponding data distribution I HR ∝P HR , λ is a hyperparameter, set to λ=5. represents the gradient penalty term derived by WGAN-LP. Represents the gradient image.

[0133] S104: Input the second feature map and the first video frame into a third neural network model for defuzzification processing to obtain a second video frame.

[0134] The third neural network model can adopt a convolutional neural network and a pyramid network. The convolutional neural network features the second feature map and the first video frame respectively to obtain feature maps of different levels corresponding to the second feature map and feature maps of different levels corresponding to the first video frame. These feature maps contain information of the image at different scales.

[0135] Through the bottom-up path and the top-down path, combined with lateral connections, a multi-scale feature pyramid is constructed. The feature maps of different levels corresponding to the second feature map and the feature maps of different levels corresponding to the first video frame are obtained. The bottom-up path generates a feature map with low resolution but high semantic information, while the top-down path enhances the detail information of the high-level feature map by upsampling. Then, the low-resolution but high-semantic information feature map and the high-level feature map with enhanced detail information are fused at multiple scales to obtain a clear and high-resolution second video frame.

[0136] The second feature map and the first video frame are processed at multiple scales by utilizing the multi-scale features in the feature pyramid, which helps to capture blur features at different scales and improve the deblurring effect.

[0137] Furthermore, an attention mechanism can be introduced into the pyramid network to better capture the key information in the image and improve the deblurring performance.

[0138] S105: Merge at least two second video frames to obtain a target video.

[0139] Since the above S102-S104 is a processing flow for one of the video frames, the initial video obtained in S101 is a video stream data, and each video frame in the initial video is super-resolution reconstructed and sharpened according to the above steps to obtain a clear and high-resolution second video frame, and multiple clear and high-resolution second video frames are merged in time series to obtain a clear and high-resolution target video.

[0140] Furthermore, when the initial video is a camera video stream, the camera captures a video frame, that is, performs super-resolution reconstruction and sharpening processing according to the above steps to obtain a clear and high-resolution second video frame, and directly displays the second video frame on the user terminal interface. Therefore, even if the user is shooting a high-speed object, real-time super-resolution reconstruction and sharpening processing can be achieved, and a clear and high-resolution target video can be directly output.

[0141] In one possible implementation, Figure 5 As shown, the main steps of inputting the first feature map into the second neural network model for super-resolution reconstruction to obtain the second feature map include S201-S206.

[0142] S201. Use a generator network to perform super-resolution reconstruction on the first feature map to obtain a third feature map.

[0143] Among them, the third feature map is the reconstructed image output by the generator network, but the third feature map is a feature map that has not been identified as true or false by the discriminator network. The second feature map also refers to the feature map output by the generator network and identified as true by the discriminator.

[0144] S202: Use a third neural network model to perform multi-scale feature extraction on the third feature map and the first video frame to obtain multi-scale features.

[0145] Among them, the third neural network model can adopt convolutional neural network and pyramid network.

[0146] For the third feature map and the first video frame, the third neural network model is used to perform multi-scale feature extraction on the third feature map output by the generator network to obtain multi-scale features, and the third neural network model is used to perform multi-scale feature extraction on the first video frame to obtain multi-scale features.

[0147] S203: Determine evaluation parameters corresponding to the third feature map based on the multi-scale features using the discriminator network.

[0148] The above evaluation parameter refers to evaluating whether the third feature map output by the generator network is a real image.

[0149] S204. If the evaluation parameter of the third feature map is the first parameter, use the first parameter as a training signal, use the training signal to update the network parameters of the generator network, and return to execute S201.

[0150] The first parameter can indicate that the third feature map output by the generator network is a false image. Then, it means that the third feature map output by the generator network cannot meet the requirements. The first parameter is fed back to the generator network to make the generator network update the network parameters. The updated generator network is used to perform super-resolution reconstruction processing to obtain a new third feature.

[0151] S205: until the evaluation parameter of the third feature map is the second parameter, the third feature map is used as the second feature map.

[0152] The first parameter can indicate that the third feature map output by the generator network is a real image. At this time, it indicates that the third feature map output by the generator network can meet the requirements, and then the third feature map that meets the requirements is used as the second feature map and input into the third neural network model for sharpening processing to obtain a clear and high-resolution second video.

[0153] In the embodiment of the present application, an unsupervised generative adversarial network for effective joint deblurring and SR is proposed, which is called VSRGAN. First, in order to make full use of the edge features in the image as auxiliary information in the deblurring process, an edge information extraction network (ERN) is proposed. After the blurred vision passes through ERN, a thin image with strong edge features (high-frequency information and dark channel information) is obtained. Secondly, in view of the problem that the complexity of the GAN model and the computational complexity cannot be balanced, a generator (RDBG) composed of an improved residual block and a residual dense block is designed. Compared with the traditional convolutional layer superposition, RDBG not only realizes the lightweight of the network structure, but also greatly improves the circulation and utilization of information. Finally, in view of the problem of low clarity and poor perceptual quality of the reconstructed image, this paper introduces a new perceptual loss function in the design of the generator loss function, which can effectively eliminate the noise in the reconstruction process and make the image more realistic and non-blurred. Compared with other state-of-the-art VSR methods, both subjective and objective evaluations are better than the state-of-the-art methods.

[0154] In order to verify the performance of the VSRGAN model, qualitative and quantitative analysis are used to prove the superiority of the video processing model. Considering that image deblurring and SR are often used as independent tasks in existing research, two groups of control experiments are established to verify the superiority of VSRGAN in deblurring and SR.

[0155] The VSRGAN and SR comparison experiment provided in this application compares VSRGAN with three popular SRs: LIIF, RAISR, and EGNet.

[0156] like Figure 6As shown, the leftmost column is the input image, column a is the image reconstructed by LR, column b is the image reconstructed by LIIF, column c is the image reconstructed by RAISR, column d is the image reconstructed by EGNet, and column e is the image reconstructed by VSRGAN.

[0157] The images reconstructed using LIIF lack texture details and have poor edge smoothness. Various noises and artifacts appear in the reconstruction results of RAISR, which may be caused by the use of the ADAM optimizer to train the discriminator. In addition, EGNet has better results, but some local colors can be found to be darker than VSRGAN. In general, the images reconstructed by VSRGAN have higher clarity and enhanced texture details, which are closer to the real scene.

[0158] The VSRGAN method is compared with the most advanced SR methods and quantitatively analyzed on the DIV2K dataset. Table 1 compares the experimental data of VSRGAN, LIIF, RAISR, and EGNet under four conditions of SR factors of 4, 8, 16, and 32. As shown in Table 1, VSRGAN achieves a good PSNR level in all cases. As the super-resolution factor gradually increases, the improvement effect of VSRGAN will gradually weaken. For example, in the DIV2K test set, when the super-resolution factor is 4, the PSNR of VSRGAN is 23.47, and the suboptimal PSNR is 0.96 higher. When the super-resolution factor is 16, VSRGAN is 0.71 higher than the suboptimal resolution. In addition, when the super-resolution factors are 8 and 16, the SSIM of VSRGAN is at the best level. When the super-resolution factor is 32, although there is a suboptimal situation, it is only 0.01 away from the optimal model. Overall, VSRGAN outperforms other state-of-the-art methods.

[0159] Table 1

[0160]

[0161] 2) Deblurring comparison experiment:

[0162] This application provides a comparative experiment between VSRGAN and defuzzification algorithms, comparing VSRGAN with three popular defuzzification algorithms: DWDN, Banet, and Ms2Net. Figure 7 As shown, column a represents blurred images, column b represents clear images after deblurring by the DWDN algorithm, column c represents clear images after deblurring by the Banet algorithm, column d represents clear images after deblurring by the Ms2Net algorithm, and column e represents clear images after deblurring by the VSRGAN algorithm.

[0163] like Figure 7As shown in Figure 1, the VSRGAN network can effectively restore the edge information of the image. Compared with the other three methods, VSRGAN is smoother and has no obvious artifacts when restoring local details of the image. Although there is still a small amount of blurring effect at the text location, the effect is better than the other three methods. The reason is that the blurring effect in this area is serious and many important image textures are lost during the blurring process.

[0164] This method is compared with the current advanced deblurring methods and quantitatively analyzed on the GoPro dataset. Table 2 compares the experimental data of VSRGAN, DWDN, Banet, and Ms2Net when the GoPro factors are 4, 8, 16, and 32 respectively.

[0165] As can be seen from Table 2, compared with the advanced methods, the VSRGAN method has obvious advantages in PSNR and SSIM indicators. This is because traditional deblurring methods have difficulty in accurately estimating blur kernels in real scenes. Banet uses convolutional neural networks to restore clear images, but since the blur kernel needs to be estimated, the deblurring effect is poor when the blur kernel estimation is inaccurate. Ms2Net uses a multi-scale convolutional neural network to gradually refine the image feature information, which slightly improves PSNR and SSIM. Since VSRGAN uses a graph edge refinement network, it works best in processing blurred images in dynamic scenes.

[0166] Table 2

[0167]

[0168] Figure 8 is a schematic diagram of the structure of a video processing device in an embodiment of the present application, such as Figure 8 As shown, the video processing device 80 provided in the embodiment of the present application mainly includes: an initial video acquisition module 81, used to acquire an initial video, wherein the initial video includes at least one first video frame; a first feature map determination module 82, used to input the first video frame into a first neural network model for the at least two first video frames to obtain a first feature map, wherein the first feature map includes high-frequency information and dark channel information; a second feature map determination module 83, used to input the first feature map into a second neural network model for super-resolution reconstruction processing to obtain a second feature map; a second video frame determination module 84, used to input the second feature map and the first video frame into a third neural network model for deblurring processing to obtain a second video frame; a target video determination module 85, used to merge at least two of the second video frames to obtain a target video.

[0169] The video processing device provided in the embodiment of the present application can execute the video processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0170] Fig. 9 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a video processing device, such as Fig. 9 As shown, the electronic device 900 includes a processor 910, a memory 920, an input device 930, and an output device 940; the number of the processor 910 in the electronic device can be one or more. Fig. 9 A processor 910 is taken as an example; the processor 910, the memory 920, the input device 930 and the output device 940 in the electronic device can be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0171] The memory 920 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video processing method in the embodiment of the present invention. The processor 910 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 920, that is, implements the video processing method provided in the embodiment of the present invention.

[0172] The memory 920 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 920 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 920 may further include a memory remotely arranged relative to the processor 910, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The input device 930 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 940 may include a display device such as a display screen.

[0174] This embodiment also provides a storage medium containing computer executable instructions, and when the computer executable instructions are executed by a computer processor, they are used to implement the video processing method provided by the embodiment of the present invention.

[0175] Of course, the computer executable instructions of a storage medium including computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the video processing method provided by any embodiment of the present invention.

[0176] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0177] It is worth noting that in the embodiment of the above-mentioned video processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0178] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0179] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire an initial video, where the initial video includes at least two first video frames; For the at least two first video frames, input the first video frames into a first neural network model to obtain a first feature map, wherein the first feature map includes high-frequency information and dark channel information; Inputting the first feature map into a second neural network model for super-resolution reconstruction to obtain a second feature map; Inputting the second feature map and the first video frame into a third neural network model for deblurring to obtain a second video frame; At least two of the second video frames are merged to obtain a target video.

2. The method according to claim 1, characterized in that The second neural network model includes a generator network and a discriminator network; Inputting the first feature map into a second neural network model for super-resolution reconstruction to obtain a second feature map, including: Using the generator network to perform super-resolution reconstruction on the first feature map to obtain a third feature map; Using the third neural network model to perform multi-scale feature extraction on the third feature map and the first video frame to obtain multi-scale features; Determining, using the discriminator network based on the multi-scale features, an evaluation parameter corresponding to the third feature map; If the evaluation parameter of the third feature map is the first parameter, using the first parameter as a training signal; Using the training signal to update the network parameters of the generator network, and returning to the step of using the generator network to perform super-resolution reconstruction processing on the first feature map to obtain a third feature map; Until the evaluation parameter of the third characteristic map is the second parameter, the third characteristic map is used as the second characteristic map.

3. The method according to claim 2, characterized in that The step of performing super-resolution reconstruction processing on the first feature map by using the generator network to obtain a third feature map includes: Using a feature extraction layer to perform shallow feature extraction on the first feature map to obtain shallow features; Extracting deep features of the first feature map using a residual network to obtain deep features; Fusing the shallow features with the deep features to obtain a combined feature; The combined features are reconstructed to obtain a third feature map.

4. The method according to claim 3, characterized in that The residual network includes a first sub-network and a second sub-network; The first sub-network includes a plurality of residual block networks connected in series, wherein the residual block network includes at least two residual blocks connected in series, and the residual block includes a plurality of convolutional layers and a parametric polymorphic linear rectification function PReLU; The second sub-network includes a plurality of residual dense networks connected in series, wherein the residual dense network includes N convolutional layers, the convolution kernels of the first N-1 convolutional layers are the same, and the convolution kernel of the last convolutional layer is different from the convolution kernels of the first N-1 convolutional layers.

5. The method according to claim 2, characterized in that The loss function of the generator network includes a perceptual loss function; The perceptual loss function is expressed by the following formula: Among them, L percep represents the perceptual loss function, I HR represents the real image, I SR represents the reconstructed image, W and H represent the feature dimensions of the specified layer in the generator network, w and h represent the horizontal and vertical coordinates of the pixel points, (I HR ) w,h represents the pixel value at coordinate (w, h) in the real image, represents the pixel value at coordinate (w, h) in the reconstructed image, represents the perceptual function of the real image at coordinate (w, h), Represents the perceptual function of the reconstructed image at coordinate (w, h).

6. The method according to claim 2, characterized in that The discriminator network includes a plurality of discriminators with the same structure, wherein the discriminator is composed of a plurality of input convolutional layers, a global average pooling layer, and an output convolutional layer connected in series; The input convolution layer includes a convolution layer, an additional Leaky layer and a PReLU activation function, and the number of filters of the multiple input convolution layers increases exponentially.

7. The method according to any one of claims 1 to 6, characterized in that: The extraction process of the dark channel information includes: Determine the blur kernel; Controlling the blur kernel to move in the first video frame according to a set rule; For the image block corresponding to the blur kernel in the first video frame, spatially rearrange according to the intensity value of each pixel point in the image block; For each pixel in the image block, multiplying the intensity value of the rearranged pixel by the intensity value corresponding to each pixel in the image block row by row; All products are added together to obtain the dark channel information of the pixel.

8. A video processing device, characterized in that: The device comprises: An initial video acquisition module, used to acquire an initial video, wherein the initial video includes at least two first video frames; A first feature map determining module is used for inputting the first video frames into a first neural network model to obtain a first feature map for the at least two first video frames, wherein the first feature map includes high frequency information and dark channel information; A second feature map determining module, used for inputting the first feature map into a second neural network model for super-resolution reconstruction to obtain a second feature map; A second video frame determination module, used for inputting the second feature map and the first video frame into a third neural network model for defuzzification processing to obtain a second video frame; The target video determination module is used to merge at least two of the second video frames to obtain a target video.

9. An electronic device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video processing method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video processing method according to any one of claims 1 to 7 is implemented.

Images