Video repair processing method, device and electronic equipment

Through deep self-attention transformation network and sliding window technology, the problem of insufficient global information utilization in video repair is solved, and a more efficient and accurate video repair effect is achieved.

CN114841867BActive Publication Date: 2024-11-15NORTHWESTERN POLYTECHNICAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210308263.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-26
Publication Date
2024-11-15
Estimated Expiration
2042-03-26

AI Technical Summary

Technical Problem

In the existing video repair technology, convolutional neural networks are difficult to effectively utilize global information when processing videos, resulting in inaccurate repair results and low efficiency.

Method used

The deep self-attention transformation network is combined with a sliding window to calculate the correlation quantization score between image feature blocks, and feature extraction and reconstruction are performed through the self-attention module and the full connection layer to replace the traditional convolutional neural network backbone network.

Benefits of technology

It significantly improves the accuracy and efficiency of video repair, can effectively utilize global information in the time domain, and improves the quality and processing speed of repair results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114841867B_ABST
    Figure CN114841867B_ABST
Patent Text Reader

Abstract

The present invention relates to a processing method, device and electronic device for video restoration. The method comprises: Step 1, establishing a neural network, and setting initialization parameters of a convolutional layer and a fully connected layer in the neural network; Step 2, obtaining a restored video corresponding to the video to be restored based on the neural network; Step 3, determining the loss function between the restored video and the standard video, and updating the initialization parameters based on the loss function; Step 4, repeatedly executing Step 2 and Step 3 until the loss function converges to a predetermined value to obtain a target neural network; Step 5, performing video restoration on the video to be restored using the target neural network. Through the present invention, the problem that the video restoration method in the related art easily leads to inaccurate restoration results and low efficiency is solved, and the technical effect of effectively improving the effect and efficiency of video restoration is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a processing method, device and electronic equipment for video restoration. Background Art

[0002] As the number of video acquisition hardware and videos grows exponentially, the demand for repairing video information defects or erasing specific targets caused by various factors is also growing. At present, the scenarios that require video repair and erasure tasks include satellite remote sensing video defogging, defect repair of historical image works, video post-production removal of pedestrians and occlusions, video watermark subtitle removal, movie special effects production, and elimination of specific human targets in specific scenes. Although the repair of incomplete information in sequence images has great research and application value in various fields, some technical difficulties of the task itself have led to its slow development and lagging application. With the continuous development of deep learning technology in the field of computer vision, the repair of defects in digital sequence images can be completed through deep neural networks.

[0003] In the prior art, a generative adversarial network video restoration method based on convolutional neural network is proposed to solve the problem of discontinuous semantic information of restoration results in video restoration. This method designs two network models: a single-frame restoration model and a sequence restoration model. The single-frame restoration model uses a single-frame stacked generator and a spatial discriminator, while the sequence restoration model uses a sequence stacked generator and a spatiotemporal discriminator. However, when the existing neural network processes the video to be restored, most of them use convolutional neural networks to extract features of effective spatiotemporal information and perform cyclic iterative restoration of the missing information. In addition, the inductive bias of the convolution operation design is to understand local visual information, and the temporal receptive field is small, which makes it impossible to effectively utilize global information, and the information is easily lost when it is transmitted between long time series.

[0004] With regard to the problem that the video restoration method in the related art easily leads to inaccurate restoration results and low efficiency, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a processing method, device and electronic device for video restoration, so as to at least solve the technical problem that the video restoration method in the related art easily leads to inaccurate restoration results and low efficiency.

[0006] According to one aspect of an embodiment of the present invention, a processing method for video restoration is provided, comprising: Step 1, establishing a neural network, and setting initialization parameters of a convolutional layer and a fully connected layer in the neural network; Step 2, obtaining a restored video corresponding to the video to be restored based on the neural network, wherein obtaining the restored video corresponding to the video to be restored based on the neural network comprises: extracting image information in the video to be restored, and converting the image information into a plurality of image feature blocks; calculating correlation quantization scores between all the image feature blocks based on a deep self-attention transformation network of a sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain the restored video; Step 3, determining a loss function between the restored video and a standard video, and updating the initialization parameters based on the loss function; Step 4, repeatedly executing Step 2 and Step 3 until the loss function converges to a predetermined value to obtain a target neural network; Step 5, performing video restoration on the video to be restored using the target neural network.

[0007] Optionally, the feature map size of the image information is a first preset multiple of the size of the video to be restored, and the number of channels of the image information is a preset number of channels.

[0008] Optionally, converting the image information into a plurality of image feature blocks includes: encoding the image information into a plurality of image feature blocks by adopting a linear transformation and an arrangement and recombination method.

[0009] Optionally, the size of the image feature block is a second preset multiple of the feature map size, and the encoding dimension of the image feature block is a preset encoding dimension.

[0010] Optionally, the loss function is expressed in the following manner: Among them, loss is the loss function, y t is the tth frame of the standard video corresponding to the repaired video, is the tth frame of the repaired video, and T is the total number of frames of the repaired video.

[0011] Optionally, updating the initialization parameters based on the loss function includes: determining the gradient of the initialization parameters corresponding to the loss function; and processing the gradient according to a gradient descent method to obtain the updated initialization parameters.

[0012] Optionally, a deep self-attention transformation network based on a sliding window calculates a correlation quantization score between all the image feature blocks, including: obtaining the correlation quantization score between any two image feature blocks in each three-dimensional window according to the deep self-attention transformation network; and determining the correlation quantization score between all the image feature blocks according to a sliding window method.

[0013] According to another aspect of an embodiment of the present invention, there is also provided a processing device for video restoration applied to any of the above methods, comprising: an establishment module, used to establish a neural network, and set initialization parameters of a convolutional layer and a fully connected layer in the neural network; a first processing module, used to obtain a restored video corresponding to the video to be restored based on the neural network, wherein obtaining the restored video corresponding to the video to be restored based on the neural network comprises: extracting image information in the video to be restored, and converting the image information into a plurality of image feature blocks; calculating the correlation quantization scores between all the image feature blocks based on a deep self-attention transformation network of a sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain the restored video; a second processing module, used to determine a loss function between the restored video and a standard video, and updating the initialization parameters based on the loss function; a third processing module, used to repeatedly execute Step 2 and Step 3 until the loss function converges to a predetermined value to obtain a target neural network; and a restoration module, used to perform video restoration on the video to be restored using the target neural network.

[0014] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the steps of any one of the above methods.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which instructions are stored, and when the instructions are executed by a processor, the steps of any one of the above methods are implemented.

[0016] In an embodiment of the present invention, Step 1, establish a neural network, and set the initialization parameters of the convolution layer and the fully connected layer in the neural network; Step 2, obtain a repaired video corresponding to the video to be repaired based on the neural network, wherein obtaining the repaired video corresponding to the video to be repaired based on the neural network includes: extracting image information in the video to be repaired, and converting the image information into multiple image feature blocks; calculating the correlation quantization scores between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain the repaired image feature blocks; mapping the repaired image feature blocks back to the video to be repaired to obtain the repaired video; Step 3, determine the loss function between the repaired video and the standard video, and update the initialization parameters based on the loss function; Step 4, repeat Step 2 and Step 3 until the loss function converges to a predetermined value to obtain the target neural network; Step 5, use the target neural network to perform video repair on the video to be repaired. That is to say, the embodiment of the present invention abandons the commonly used convolutional neural network as the backbone network, and adopts a deep self-attention transformation network composed entirely of self-attention modules and fully connected layers to perform feature extraction and effective information search and reconstruction, and adds a sliding window to improve efficiency, thereby solving the problem that the video restoration method in the related technology easily leads to inaccurate restoration results and low efficiency, and achieves the technical effect of effectively improving the effect and efficiency of video restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 A flowchart of a video repair processing method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a deep self-attention transformation network provided for an optional embodiment of the present invention;

[0020] Figure 3 A schematic diagram of a neural network provided for an optional embodiment of the present invention;

[0021] Figure 4 A schematic diagram of a video restoration processing device provided by an embodiment of the present invention;

[0022] Figure 5 Test result graph. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second" and the like in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to limit a specific order.

[0025] In order to repair a video with incomplete information, an embodiment of the present invention provides a video repair processing method. Figure 1 A flowchart of a video repair processing method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the video repair processing method provided in the embodiment of the present application includes the following steps:

[0026] Step 1: Establish a neural network and set the initialization parameters of the convolutional layer and the fully connected layer in the neural network;

[0027] The above initialization parameters include but are not limited to the number of fully connected layers (neurons), the number of levels, the size of the convolution kernel, etc.

[0028] It should be noted that the structure of the above neural network at least includes: a single-layer two-dimensional convolution encoder, a two-dimensional feature image block encoding, a video restoration module, a single-layer two-bit convolution, a two-dimensional feature image block inverse encoding, and a single-layer two-dimensional convolution decoder, wherein the video restoration module includes a sliding window deep self-attention transformation calculation and a single-layer two-bit convolution connected thereto. The single-layer two-dimensional convolution encoder is used to extract image information from the video to be restored; the two-dimensional feature image block encoding is connected to the single-layer two-dimensional convolution encoder to convert the image information into multiple image feature blocks; the above video restoration module is connected to the two-dimensional feature image block encoding to calculate the correlation quantization score between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encode the image feature block according to the correlation quantization score and the initialization parameter to obtain the restored image feature block; the two-dimensional feature image block inverse encoding and the single-layer two-dimensional convolution decoder are used to map the restored image feature block back to the video to be restored to obtain the restored video. In addition, one side of the single-layer two-bit convolution is connected to the video restoration module, and the other side is connected to the two-dimensional feature image block inverse encoding.

[0029] The above-mentioned single-layer two-dimensional convolution encoder is an encoder composed of a single-layer two-dimensional convolution; the above-mentioned single-layer two-dimensional convolution decoder is a decoder composed of a single-layer two-dimensional convolution. In addition, the two-dimensional feature image block inverse encoding and the single-layer two-dimensional convolution decoder operation composition are opposite to the input and output of the two-dimensional feature image block encoding and the single-layer two-dimensional convolution encoder.

[0030] Step 2, obtaining a restored video corresponding to the video to be restored based on a neural network, wherein obtaining a restored video corresponding to the video to be restored based on a neural network includes: extracting image information from the video to be restored, and converting the image information into multiple image feature blocks; calculating the correlation quantization scores between all image feature blocks based on a deep self-attention transformation network of a sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and initialization parameters to obtain restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain a restored video;

[0031] In an optional implementation, the video to be repaired may be input into a neural network, which processes the video to be repaired and outputs a repaired video corresponding to the video to be repaired.

[0032] Figure 2 A schematic diagram of a deep self-attention transformation network provided for an optional embodiment of the present invention, such as Figure 2 As shown in the figure, the input sequence is positionally encoded and enters the encoder, and then is sent to the multi-head attention module, and then after a residual connection and regularization, it is finally sent to the residual connection module composed of the forward fully connected layer. This process is a complete deep self-attention transform network encoding.

[0033] For a deep self-attention transformer network encoding process, the focus is on the calculation of the attention module (Attention), which can be expressed as follows:

[0034]

[0035] Among them, Q, K, and V are linear mappings from the input sequence, Q represents the query, K and V represent the key-value pairs, and d k represents the dimension of K, and softmax represents the normalized exponential function. The matrix output by the normalized exponential function is the attention distribution matrix. Each value in the matrix corresponds to the attention information between any two positions of Q and K. In the video restoration task, it can represent the degree of association between two image feature blocks in different spatiotemporal positions. Multi-head attention is to perform multiple sets of different linear mappings on the same input to calculate self-attention separately, and then concatenate their outputs into a designed size, which can make the model pay attention to attention information of different scales.

[0036] For video restoration tasks, the input of the original deep self-attention transform network is usually a stack of multi-frame coded feature maps in the channel dimension, which makes the input sequence too long. The existence of secondary computational complexity makes the original model computational load too high and the long sequence processing capability is poor. To solve this problem, the present invention proposes to add a sliding window method to limit the self-attention calculation between image feature blocks within a three-dimensional window, and establish information exchange between windows through sliding windows to achieve the purpose of reducing computational complexity and improving efficiency.

[0037] Specifically, the feature size of the encoded multi-frame image to be repaired is T×h×w×c, where T represents the total number of frames; T×h×w represents the number of feature image feature blocks; and c represents the encoding dimension of each image feature block. By designing P×P three-dimensional windows and using them to segment the original input, each window contains image feature blocks. At this time, the self-attention calculation between blocks changes from the calculation between T×h×w to The calculation complexity between windows is greatly reduced. In addition, in order to enhance the information exchange between windows, after a calculation using the regular window self-attention module, a calculation using the self-attention module that slides the window is connected. Set the sliding size to That is, the position of the window segmentation feature block is changed according to the size. A conventional window self-attention module and a window sliding self-attention module together constitute a sliding window deep self-attention transformation network, which can be used to implement a complete sliding window based deep self-attention transformation calculation. Finally, just replace the above ordinary self-attention module with the window-containing self-attention module proposed here.

[0038] Step 3: Determine the loss function between the restored video and the standard video, and update the initialization parameters based on the loss function;

[0039] Step 4: Repeat Step 2 and Step 3 until the loss function converges to a predetermined value and the target neural network is obtained.

[0040] Optionally, after updating the initialization parameters based on the loss function, since the initialization parameters of the neural network have been corrected, the repaired video corresponding to the video to be repaired can be obtained based on the neural network again, and then the loss function between the repaired video and the standard video can be determined, and the initialization parameters can be updated based on the loss function. This is repeated multiple times until the loss function converges to a predetermined value, and the neural network is used as the target neural network, wherein the target neural network is used to perform video repair on the video to be repaired.

[0041] It should be noted that the initialization parameters are updated through the loss function to minimize the loss function value. Through the above implementation, the initialization parameters in the neural network can be continuously updated to obtain a new neural network, and the new neural network can be continuously trained until the loss function between the restored video and the standard video converges to a predetermined value, thereby obtaining a target neural network that can perform video restoration on the video to be restored.

[0042] The above-mentioned preset value can be flexibly set according to the needs of the application scenario and is not limited here.

[0043] Step 5: Use the target neural network to perform video restoration on the video to be restored.

[0044] In an embodiment of the present invention, Step 1, establish a neural network, and set the initialization parameters of the convolution layer and the fully connected layer in the neural network; Step 2, obtain a repaired video corresponding to the video to be repaired based on the neural network, wherein obtaining the repaired video corresponding to the video to be repaired based on the neural network includes: extracting image information in the video to be repaired, and converting the image information into multiple image feature blocks; calculating the correlation quantization scores between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain the repaired image feature blocks; mapping the repaired image feature blocks back to the video to be repaired to obtain the repaired video; Step 3, determine the loss function between the repaired video and the standard video, and update the initialization parameters based on the loss function; Step 4, repeat Step 2 and Step 3 until the loss function converges to a predetermined value to obtain the target neural network; Step 5, use the target neural network to perform video repair on the video to be repaired. That is to say, the embodiment of the present invention abandons the commonly used convolutional neural network as the backbone network, and adopts a deep self-attention transformation network composed entirely of self-attention modules and fully connected layers to perform feature extraction and effective information search and reconstruction, and adds a sliding window to improve efficiency, thereby solving the problem that the video restoration method in the related technology easily leads to inaccurate restoration results and low efficiency, and achieves the technical effect of effectively improving the effect and efficiency of video restoration.

[0045] In an optional implementation, the feature map size of the above-mentioned image information is a first preset multiple of the size of the video to be restored, and the number of channels of the image information is a preset number of channels.

[0046] It should be noted that the first preset multiple includes but is not limited to etc.; preset channel numbers include but are not limited to 128, 256, etc.

[0047] In an optional implementation, converting the image information into a plurality of image feature blocks includes: encoding the image information into a plurality of image feature blocks by linear transformation and permutation and recombination.

[0048] In the above implementation, the image information may be encoded into a plurality of image feature blocks by means of linear transformation and permutation and recombination, thereby realizing the conversion of the image information into the image feature blocks.

[0049] In an optional implementation, the size of the image feature block is a feature map size of a second preset multiple, and the encoding dimension of the image feature block is a preset encoding dimension.

[0050] It should be noted that the second preset multiple includes but is not limited to etc.; preset encoding dimensions include but are not limited to 180 etc.

[0051] In an optional implementation, the above loss function is expressed in the following manner: Among them, loss is the loss function, y t is the tth frame of the standard video corresponding to the repaired video, is the tth frame of the repaired video, and T is the total number of frames of the repaired video.

[0052] In the above implementation, the loss function can be used to calculate the loss function value between the restored video and the standard video, and then the corresponding parameters in the neural network can be updated, so as to facilitate the loss function to converge to a predetermined value.

[0053] In an optional implementation, updating the initialization parameters based on the loss function includes: determining the gradient of the initialization parameters corresponding to the loss function; and processing the gradient according to a gradient descent method to obtain updated initialization parameters.

[0054] In the above implementation, the gradient of the loss function with respect to all trainable parameters is calculated, and all parameters are updated using the gradient descent method to minimize the loss function value.

[0055] In an optional embodiment, a deep self-attention transformation network based on a sliding window calculates the correlation quantization scores between all image feature blocks, including: obtaining the correlation quantization scores between any two image feature blocks in each three-dimensional window according to the deep self-attention transformation network; and determining the correlation quantization scores between all image feature blocks according to a sliding window method.

[0056] In the above implementation, the correlation quantization score between any two image feature blocks calculated by the deep self-attention transformation network is restricted to a three-dimensional window, and the correlation quantization score between all image feature blocks is calculated by a sliding window method, which can achieve the purpose of reducing computational complexity and improving efficiency.

[0057] The following is an example of video restoration of a video target segmentation dataset (Denly-Annotated VIdeo Segmentation, DAVIS) video database to illustrate this embodiment. The specific implementation method includes the following steps:

[0058] Step 1: If Figure 3 As shown in the figure, a neural network is constructed, and the parameters of all convolutional and fully connected layers are initialized with random numbers. The encoder output feature map size composed of a single-layer two-dimensional convolution is 1 / 4 of the input, and the number of channels is 128. The two-dimensional feature image block encoding uses linear transformation and permutation and reorganization to encode the feature map into an image feature block. The output size is 1 / 2 of the input, and the feature block encoding dimension is 180. The number of video restoration modules can be flexibly adjusted. Taking 4 as an example, each module consists of 3 calculations based on the sliding window deep self-attention transformation, and connects a single-layer convolution operation and a residual connection to improve the efficiency and accuracy of network training. The image feature block inverse encoding and decoder operation composition are opposite to the image feature block encoding and the encoder input and output, and the high-dimensional repaired feature map can be mapped back to the image layer.

[0059] Step 2: Forward propagation: For an input video to be repaired, the image information in the red, green and blue three-channel video frame is extracted through a single-layer two-dimensional convolutional encoder and two-dimensional feature image block encoding, and a 180-channel high-dimensional feature block is output.

[0060] Then the sliding window deep self-attention calculation in the video restoration module will obtain the correlation quantization score between any two image feature blocks in each 3D window, and then through the sliding window method, the correlation quantization score between all input image feature blocks will be known. Based on these scores, the parameters of the fully connected network will re-encode the features and output the restored features.

[0061] Finally, after the image feature block inverse encoding and decoder, the repaired high-dimensional features are mapped back to the three-channel video frame to obtain the final repaired video.

[0062] Step 3: Back propagation. Let To repair the video for T frames, To output T frames of repaired video, Y1 T ={y1,y2,L y T} is T frames of real defect-free video, and the video restoration method is f(x). Then the forward propagation of the restoration process can be expressed as follows:

[0063]

[0064] and With Y1 T The loss function calculation between can be expressed as follows:

[0065]

[0066] The loss function between the output of the forward propagation and the real intact video frame is calculated, and its gradient relative to all trainable parameters is calculated, and all parameters are updated using the gradient descent method to minimize the loss function value.

[0067] Step 4: Repeat steps 2 and 3 for all video samples in the database until the loss function converges to a relatively stable value.

[0068] Step 5: Use the forward propagation of the video restoration algorithm trained in the above steps to restore the DAVIS test set. The average peak signal-to-noise ratio and structural similarity between the real video and the real video are calculated, reaching 31.01 and 95.77% respectively.

[0069] It should be noted that the above-mentioned embodiments of the present invention can achieve the following beneficial effects:

[0070] (1) The deep self-attention transformer network using a sliding window can effectively extract features from the temporal global information of the video input and search and match valid relevant information.

[0071] (2) The video restoration algorithm using the deep self-attention transformation network with sliding window significantly improves the accuracy and efficiency of the video restoration results.

[0072] The embodiment of the present invention further provides a processing device for video restoration applied to any of the above methods. Figure 4 A schematic diagram of a video restoration processing device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the video restoration processing device provided in the embodiment of the present application includes: a creation module 41, a first processing module 43, a second processing module 45, a third processing module 47 and a restoration module 49. The video restoration processing device is described in detail below.

[0073] The establishment module 41 is used to establish a neural network and set the initialization parameters of the convolution layer and the fully connected layer in the neural network; the first processing module 43 is connected to the above-mentioned establishment module 41, and is used to obtain the restored video corresponding to the video to be restored based on the neural network, wherein the restoration video corresponding to the video to be restored based on the neural network includes: extracting image information in the video to be restored, and converting the image information into multiple image feature blocks; calculating the correlation quantization score between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization score and the initialization parameters to obtain the restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain the restored video; the second processing module 45 is connected to the above-mentioned first processing module 43, and is used to determine the loss function between the restored video and the standard video, and update the initialization parameters based on the loss function; the third processing module 47 is connected to the above-mentioned second processing module 45, and is used to repeatedly execute Step 2 and Step 3 until the loss function converges to a predetermined value to obtain the target neural network; the restoration module 49 is connected to the above-mentioned third processing module 47, and is used to perform video restoration on the video to be restored using the target neural network.

[0074] In the above embodiment, the processing device for video repair can use the establishment module 41 to establish a neural network, and set the initialization parameters of the convolution layer and the fully connected layer in the neural network; then use the first processing module 43 to obtain the repaired video corresponding to the video to be repaired based on the neural network, wherein the repaired video corresponding to the video to be repaired based on the neural network includes: extracting image information in the video to be repaired, and converting the image information into multiple image feature blocks; calculating the correlation quantization score between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization score and the initialization parameters to obtain the repaired image feature blocks; mapping the repaired image feature blocks back to the video to be repaired to obtain the repaired video; then using the second processing module 45 to determine the loss function between the repaired video and the standard video, and updating the initialization parameters based on the loss function; the third processing module 47 is used to repeatedly execute Step 2 and Step 3 until the loss function converges to a predetermined value to obtain the target neural network; finally, the video to be repaired is repaired by using the target neural network through the repair module 49. That is to say, the embodiment of the present invention abandons the commonly used convolutional neural network as the backbone network, and adopts a deep self-attention transformation network composed entirely of self-attention modules and fully connected layers to perform feature extraction and effective information search and reconstruction, and adds a sliding window to improve efficiency, thereby solving the problem that the video restoration method in the related technology easily leads to inaccurate restoration results and low efficiency, and achieves the technical effect of effectively improving the effect and efficiency of video restoration.

[0075] It should be noted here that the above-mentioned establishment module 41, first processing module 43, second processing module 45, third processing module 47 and repair module 49 correspond to steps Step 1 to Step 5 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment.

[0076] Optionally, the feature map size of the above image information is a first preset multiple of the size of the video to be restored, and the number of channels of the image information is a preset number of channels.

[0077] It should be noted that the first preset multiple includes but is not limited to etc.; preset channel numbers include but are not limited to 128, 256, etc.

[0078] Optionally, the first processing module 43 includes: an encoding unit, configured to encode the image information into a plurality of image feature blocks by adopting a linear transformation and permutation and reorganization method.

[0079] In the above implementation, the image information may be encoded into a plurality of image feature blocks by means of linear transformation and permutation and recombination, thereby realizing the conversion of the image information into the image feature blocks.

[0080] Optionally, the size of the image feature block is a feature map size of a second preset multiple, and the encoding dimension of the image feature block is a preset encoding dimension.

[0081] It should be noted that the second preset multiple includes but is not limited to etc.; preset encoding dimensions include but are not limited to 180 etc.

[0082] Optionally, the above loss function is expressed in the following way: Among them, loss is the loss function, y t is the tth frame of the standard video corresponding to the repaired video, is the tth frame of the repaired video, and T is the total number of frames of the repaired video.

[0083] In the above implementation, the loss function can be used to calculate the loss function value between the restored video and the standard video, and then the corresponding parameters in the neural network can be updated, so as to facilitate the loss function to converge to a predetermined value.

[0084] Optionally, the second processing module 45 includes: a determination unit, used to determine the gradient of the initialization parameter corresponding to the loss function; and a first processing unit, used to process the gradient according to the gradient descent method to obtain the updated initialization parameter.

[0085] In the above implementation, the gradient of the loss function with respect to all trainable parameters is calculated, and all parameters are updated using the gradient descent method to minimize the loss function value.

[0086] Optionally, the above-mentioned first processing module 43 includes: a second processing unit, used to obtain the correlation quantization score between any two image feature blocks in each three-dimensional window according to the deep self-attention transformation network; a third processing unit, used to determine the correlation quantization score between all image feature blocks according to the sliding window method.

[0087] In the above implementation, the correlation quantization score between any two image feature blocks calculated by the deep self-attention transformation network is restricted to a three-dimensional window, and the correlation quantization score between all image feature blocks is calculated by a sliding window method, which can achieve the purpose of reducing computational complexity and improving efficiency.

[0088] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the steps of any one of the above methods.

[0089] Optionally, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the program: Step 1, establishing a neural network, and setting initialization parameters of a convolutional layer and a fully connected layer in the neural network; Step 2, obtaining a repaired video corresponding to the video to be repaired based on the neural network, wherein obtaining a repaired video corresponding to the video to be repaired based on the neural network includes: extracting image information in the video to be repaired, and converting the image information into multiple image feature blocks; calculating correlation quantization scores between all image feature blocks based on a deep self-attention transformation network of a sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain repaired image feature blocks; mapping the repaired image feature blocks back to the video to be repaired to obtain the repaired video; Step 3, determining a loss function between the repaired video and the standard video, and updating the initialization parameters based on the loss function; Step 4, repeatedly executing Step 2 and Step 3 until the loss function converges to a predetermined value to obtain a target neural network; Step 5, performing video repair on the video to be repaired using the target neural network.

[0090] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which instructions are stored, and when the instructions are executed by a processor, the steps of any of the above methods are implemented.

[0091] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, and / or located in any one of the mobile terminals in a mobile terminal group, and the computer-readable storage medium includes a stored program.

[0092] Optionally, when the program is running, the device where the computer-readable storage medium is located is controlled to perform the following functions: Step 1, establish a neural network, and set the initialization parameters of the convolution layer and the fully connected layer in the neural network; Step 2, obtain the repaired video corresponding to the video to be repaired based on the neural network, wherein the repaired video corresponding to the video to be repaired based on the neural network includes: extracting image information in the video to be repaired, and converting the image information into multiple image feature blocks; calculating the correlation quantization score between all image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization score and the initialization parameters to obtain the repaired image feature blocks; mapping the repaired image feature blocks back to the video to be repaired to obtain the repaired video; Step 3, determine the loss function between the repaired video and the standard video, and update the initialization parameters based on the loss function; Step 4, repeat Step 2 and Step 3 until the loss function converges to a predetermined value to obtain the target neural network; Step 5, use the target neural network to perform video repair on the video to be repaired.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A video restoration processing method, characterized in that: include: Step 1: Establish a neural network and set the initialization parameters of the convolutional layer and the fully connected layer in the neural network; Step 2, obtaining a restored video corresponding to the video to be restored based on the neural network, wherein obtaining a restored video corresponding to the video to be restored based on the neural network includes: extracting image information from the video to be restored, and converting the image information into multiple image feature blocks; calculating the correlation quantization scores between all the image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain the restored video; The sliding window-based deep self-attention transformation network calculates the correlation quantization scores between all the image feature blocks, including: Obtaining the correlation quantization score between any two image feature blocks in each three-dimensional window according to the deep self-attention transformation network; Determine the correlation quantization scores between all the image feature blocks in a sliding window manner; Step 3, determining the loss function between the restored video and the standard video, and updating the initialization parameters based on the loss function; Step 4, repeat Step 2 and Step 3 until the loss function converges to a predetermined value, and obtains the target neural network; Step 5: Use the target neural network to perform video restoration on the video to be restored.

2. The processing method according to claim 1, characterized in that: The feature map size of the image information is a first preset multiple of the size of the video to be restored, and the number of channels of the image information is a preset number of channels.

3. The processing method according to claim 1, characterized in that: The image information is converted into a plurality of image feature blocks, including: The image information is encoded into a plurality of image feature blocks by adopting a linear transformation and arrangement and recombination method.

4. The processing method according to claim 1, characterized in that: The size of the image feature block is a second preset multiple of the feature map size, and the encoding dimension of the image feature block is a preset encoding dimension.

5. The processing method according to claim 1, characterized in that: The loss function is expressed as follows: Among them, loss is the loss function, y t is the tth frame of the standard video corresponding to the repaired video, is the tth frame of the repaired video, and T is the total number of frames of the repaired video.

6. The processing method according to claim 1, characterized in that: Updating the initialization parameter based on the loss function includes: Determine the gradient of the initialization parameter corresponding to the loss function; The gradient is processed according to the gradient descent method to obtain the updated initialization parameter.

7. A processing device for video restoration applied to the method according to any one of claims 1 to 6, characterized in that: include: Establishing a module for establishing a neural network and setting initialization parameters of a convolutional layer and a fully connected layer in the neural network; A first processing module is used to obtain a restored video corresponding to the video to be restored based on the neural network, wherein obtaining the restored video corresponding to the video to be restored based on the neural network includes: extracting image information in the video to be restored, and converting the image information into multiple image feature blocks; calculating the correlation quantization scores between all the image feature blocks based on the deep self-attention transformation network of the sliding window, and re-encoding the image feature blocks according to the correlation quantization scores and the initialization parameters to obtain restored image feature blocks; mapping the restored image feature blocks back to the video to be restored to obtain the restored video; A second processing module, configured to determine a loss function between the restored video and a standard video, and update the initialization parameter based on the loss function; A third processing module is used to repeatedly execute Step 2 and Step 3 until the loss function converges to a predetermined value to obtain a target neural network; A repair module is used to perform video repair on the video to be repaired using the target neural network.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.