Large-area defect video restoration method and device based on frequency domain fusion

Abstract

The invention discloses a large-area defect video restoration method and device based on frequency domain fusion, relates to the technical field of video processing, and aims to solve the problems that a defect video restoration method at the present stage is mainly limited to a small-area defect scene, the restoration capability of large-area defect video content is insufficient, and a reasonable visual restoration result is difficult to generate. An effective solution is provided, and the method comprises the following steps: firstly, obtaining a defective video frame sequence, and carrying out down-sampling on the video frame sequence; next, global information modeling is carried out on the defect video frame after down-sampling by using a stacked frequency domain fusion residual block, and the frequency domain fusion residual block is formed by sequentially connecting two adaptive frequency domain cross fusion modules; secondly, optimizing the time consistency among a plurality of frames by using a stacked time Transform module; and finally, performing up-sampling to reconstruct a video frame, and obtaining a final restored video. According to the method, a video restoration effect which is visually reasonable and smooth and natural in content can be generated in a large-range defect area.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method and device for repairing large-area defective video based on frequency domain fusion. Background Art

[0002] Video restoration technology refers to the process of filling and completing the defective areas in the video frame through algorithms, so that the repaired defective areas are consistent with the surrounding content and maintain the smoothness of the video, thereby restoring the integrity and naturalness of the video. This technology is widely used in post-production video editing, film restoration, video transmission damage repair, virtual reality and other fields.

[0003] At present, the research and application of video restoration technology mainly focus on methods based on deep learning. Some video restoration methods that combine optical flow propagation and Transformer have achieved remarkable results in the restoration task of small-area defects. These methods can generate a completed area that is highly coordinated with the surrounding content and maintain the temporal consistency between video frames to a certain extent. However, these methods mainly repair small-area defect areas, and their ability to repair large-area defects is still insufficient. In the small-area defect scene, the model can make full use of the rich contextual information around the defect area and reasonably predict and generate the defect content through the spatial and temporal correlation. However, when the defect area is large, the available contextual information is significantly reduced, and the model lacks sufficient reference data, resulting in unreasonable restoration results. At present, there is still a lack of effective solutions for video restoration technology for large-area defect areas. Unlike general small-area defect restoration, the challenge of large-area defect restoration lies in how to effectively fill the large area of ​​missing areas and keep the restored content visually reasonable. Although it is not required to be completely accurate with the real data, the repaired area needs to have a reasonable structure and texture in space and maintain consistency with adjacent frames in the time domain to ensure that the audience has a smooth visual experience. Therefore, large-area defective video repair has become a difficult problem that needs to be solved in the field of video processing, and more innovative and targeted technologies are needed to deal with this problem. Summary of the invention

[0004] The purpose of the present invention is to provide a method and device for repairing large-area defective videos based on frequency domain fusion, which can generate a visually reasonable video repair effect with smooth and natural content in a large-scale defective area.

[0005] The present invention adopts the following technical solution:

[0006] On the one hand, a large area defective video restoration method based on frequency domain fusion includes:

[0007] Acquire a defective video frame sequence, and downsample a plurality of continuous video frames in the video frame sequence to obtain a corresponding first feature;

[0008] The first feature is input into the stacked frequency domain fusion residual block, and the stacked frequency domain fusion residual block is used to perform global information modeling on the first feature; each layer of the frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence; each frequency domain cross fusion module receives the input feature, and decomposes the input feature into local features and global features along the channel dimension based on the channel ratio; the local feature is input into the local branch, and the local branch uses a 3×3 ordinary convolution operation on the local feature to extract the local spatial feature; the global feature is input into the global branch, and the global branch converts the global feature from the spatial domain to the frequency domain through an adaptive frequency domain convolution filter module, strengthens and learns the global feature, fully captures the global context information, and then restores it to the spatial domain; the local branch and the global branch complement each other through the cross fusion mechanism, and splice in the channel dimension to generate the output feature; wherein the two frequency domain cross fusion modules are respectively the first frequency domain cross fusion module and the second frequency domain cross fusion module, the input feature of the first frequency domain cross fusion module is the first feature, the input feature of the second frequency domain cross fusion module is the output feature of the first frequency domain cross fusion module, and the output feature of the second frequency domain cross fusion module is the second feature;

[0009] The second feature is input into the stacked temporal Transformer module, and the temporal consistency between frames is optimized through the stacked temporal Transformer module to obtain the third feature;

[0010] The third feature is upsampled to reconstruct the video frame and obtain the final repaired video.

[0011] Preferably, each frequency domain cross fusion module receives input features and decomposes the input features into local features and global features along the channel dimension based on the channel ratio, specifically including:

[0012] Each frequency domain cross fusion module receives input features ;in, , and Represent the height, width and number of channels of the feature respectively, represents the set of real numbers;

[0013] Input features Decompose into local features along the channel dimension and global features ;in, Indicates the ratio of channels allocated to global features.

[0014] Preferably, the processing process and cross-fusion process of the local branch and the global branch are as follows:

[0015] ;

[0016] ;

[0017] ;

[0018] in, Represents the output features of the local branch; represents the activation function; represents normalization; Represents a 3×3 normal convolution operation; Represents the output features of the global branch; Represents normalization and activation function; represents an adaptive frequency domain convolution filter that converts features into the frequency domain to capture global context information; represents the output features; Indicates that and Perform concatenation on the channel dimension.

[0019] Preferably, the processing process of the adaptive frequency domain convolution filter is as follows:

[0020] Adaptive frequency domain convolution filter receives global features First, the fast Fourier transform (FFT) is used to convert the global features into the corresponding frequency domain representation, and the imaginary part and the real part of the complex result are concatenated to obtain the frequency domain features. , and then the frequency domain features are adjusted by adaptively learning the filter weights Update the global information, expressed as:

[0021] ;

[0022] in, Represents the frequency domain features after global information update; represents element-wise product; Represents the adaptive filter weights learned from frequency domain features.

[0023] Preferably, the learning process of the adaptive filtering weights is as follows:

[0024] First, a 1×1 group convolution is used for feature mapping, then the ReLU activation function is applied, and finally another 1×1 group convolution is used to generate the filter weights so that the frequency domain features can complete the global information update; the updated frequency domain features Split into real and imaginary parts along the channel dimension, reconvert back to the representation of the integer, and finally convert back to the spatial domain features through the inverse fast Fourier transform IFFT .

[0025] Preferably, the processing process of the temporal Transformer module is as follows:

[0026] The temporal Transformer module receives the second feature processed by the frequency domain fusion residual block, divides each feature map in the second feature into non-overlapping cubes along the height and width dimensions, performs multi-head self-attention operations in these cube areas, and obtains the third feature .

[0027] On the other hand, a large area defective video restoration device based on frequency domain fusion includes:

[0028] A downsampling module, used to obtain a defective video frame sequence, and downsample a plurality of continuous video frames in the video frame sequence to obtain a corresponding first feature;

[0029] The frequency domain fusion residual processing module is used to input the first feature into the stacked frequency domain fusion residual blocks, and use the stacked frequency domain fusion residual blocks to perform global information modeling on the first feature; each layer of the frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence; each frequency domain cross fusion module receives the input feature, and decomposes the input feature into local features and global features along the channel dimension based on the channel ratio; the local feature is input into the local branch, and the local branch uses a 3×3 ordinary convolution operation to extract local spatial features from the local feature; the global feature is input into the global branch, and the global branch uses an adaptive frequency domain convolution filter model The block converts the global features from the spatial domain to the frequency domain, strengthens and learns the global features, fully captures the global context information, and then restores it to the spatial domain; the local branch and the global branch achieve information complementarity through the cross-fusion mechanism, and splice in the channel dimension to generate output features; wherein the two frequency domain cross-fusion modules are respectively the first frequency domain cross-fusion module and the second frequency domain cross-fusion module, the input feature of the first frequency domain cross-fusion module is the first feature, the input feature of the second frequency domain cross-fusion module is the output feature of the first frequency domain cross-fusion module, and the output feature of the second frequency domain cross-fusion module is the second feature;

[0030] A time consistency processing module, used for inputting the second feature into a stacked time Transformer module, optimizing the time consistency between frames through the stacked time Transformer module, and obtaining a third feature;

[0031] The upsampling module is used to upsample the third feature to reconstruct the video frame to obtain the final restored video.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The present invention realizes efficient global fusion modeling of spatial information through a frequency domain cross fusion module of stacked frequency domain fusion residual blocks;

[0034] (2) The adaptive frequency domain convolution filter of the present invention can convert spatial features into frequency domain for operation, and has the ability to capture global information in large-area defect scenes, while strengthening the effective information expression of features;

[0035] (3) The temporal Transformer module of the present invention performs attention calculation on the features in the multi-frame time dimension, which can effectively optimize the temporal consistency of the restoration results and ensure the coherence of the video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of a method for repairing a large-area defective video based on frequency domain fusion according to an embodiment of the present invention;

[0037] Figure 2 It is a structural schematic diagram of a frequency domain cross-fusion module according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of the structure of an adaptive frequency domain convolution filter according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the temporal Transformer module and cubic window division of an embodiment of the present invention;

[0040] Figure 5 It is a structural schematic diagram of a large-area defective video restoration device based on frequency domain fusion according to an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0043] like Figure 1As shown, the present embodiment is a method for repairing a large-area defective video based on frequency domain fusion, comprising the following steps: obtaining a video frame with a large area defect, and performing a downsampling operation on it to obtain a first feature Z1; then, a frequency domain fusion residual block stacked with L layers receives the feature Z1, and performs efficient global information modeling. The frequency domain fusion residual block is connected in sequence by two frequency domain cross-fusion modules, and a residual connection is constructed, and the frequency domain fusion residual block outputs the second feature Z2. Then, a time Transformer module stacked with N layers optimizes the temporal consistency of feature Z2 to obtain a third feature Z3; finally, an upsampling operation is performed on feature Z3 to reconstruct the video frame to obtain the final repaired video.

[0044] The stacked L layers of frequency domain fusion residual blocks are specifically L layers of frequency domain fusion residual blocks connected sequentially.

[0045] The following is a detailed description of the process steps of the large-area defective video restoration method based on frequency domain fusion.

[0046] First, for the defective video sequence , use convolutional neural network to downsample it and get the first feature of channel c .in, and Represent the spatial resolution and the number of channels, respectively. Indicates the first frame.

[0047] Next, the first feature Enter the stacked frequency domain fusion residual block.

[0048] The frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence. The output processed by the two frequency domain cross fusion modules is added to the input of the residual block through residual connection to complete the fusion and transmission of features.

[0049] like Figure 2 The figure shows the structure of the frequency domain cross fusion module. The frequency domain cross fusion module adopts the construction idea of ​​local-global two-way branch cross fusion, combining the local convolution operation in the spatial domain with the global modeling capability in the frequency domain, so as to effectively capture and learn the residual feature information under large-area invalid features. The frequency domain cross fusion module receives the input feature ,in and Denote the spatial resolution and number of channels respectively. First, the input features are decomposed into local features along the channel dimension and global features , Indicates the channel ratio assigned to global features. Local features Enter the local branch to capture local spatial features, global features Enter the global branch to capture long-range global context information. The two branches complement each other through the cross-fusion mechanism, enhance the expression of effective features, and splice in the channel dimension to finally generate output features. Specifically, the local branch A 3×3 ordinary convolution operation is used to extract local spatial features, and the global branch uses an adaptive frequency domain convolution filter module to Convert from the spatial domain to the frequency domain, strengthen and learn global features, fully capture global context information, and then restore to the spatial domain. The calculation process and cross-fusion method of the two branches are described as follows:

[0050] ;

[0051] ;

[0052] ;

[0053] in, Represents the output features of the local branch; represents the activation function; represents normalization; Represents a 3×3 normal convolution operation; Represents the output features of the global branch; Represents normalization and activation function; represents an adaptive frequency domain convolution filter that converts features into the frequency domain to capture global context information; represents the output features; Indicates that and Perform concatenation on the channel dimension.

[0054] It should be noted that the two frequency domain cross-fusion modules can be the first frequency domain cross-fusion module and the second frequency domain cross-fusion module respectively, the input feature of the first frequency domain cross-fusion module is the first feature Z1, the input feature of the second frequency domain cross-fusion module is the output feature of the first frequency domain cross-fusion module, and the output feature of the second frequency domain cross-fusion module is the second feature Z2.

[0055] In this embodiment, the structure of the adaptive frequency domain convolution filter is as follows: Figure 3 As shown in Figure 1, the adaptive frequency domain convolution filter converts the features from the spatial domain to the frequency domain for efficient global information modeling, and then converts the updated frequency domain features back to the spatial domain features. Specifically, the second feature receives the global feature First, the fast Fourier transform (FFT) is used to convert it into the corresponding frequency domain representation. Since the calculation result of the fast Fourier transform contains complex numbers, in order to ensure that the input and output of this module are real numbers, the imaginary part and the real part of the complex result are spliced ​​to obtain the frequency domain feature. , and then the frequency domain features are adjusted by adaptively learning the filter weights To update the global information, this process is expressed as:

[0056] ;

[0057] in, is the frequency domain feature after global information update, Represents element-wise product. is the adaptive filter weight learned from frequency domain features. Filter weight The learning of is achieved through the following steps: first, a 1×1 group convolution is used for feature mapping, then the ReLU activation function is applied, and finally another 1×1 group convolution is used to generate the filter weights. Through the above process, the frequency domain features complete the global information update. The updated frequency domain features Split into real and imaginary parts along the channel dimension, reconvert back to the representation of the integer, and finally convert back to the spatial domain features through inverse fast Fourier transform .

[0058] After all the frequency domain fusion residual blocks, the second feature is obtained , then enter the N-layer stacked temporal Transformer module.

[0059] The N-layer stacked temporal Transformer module is specifically a temporal Transformer module with N layers connected in sequence.

[0060] like Figure 4 The figure shows a schematic diagram of the temporal Transformer module and the cubic window division. Its self-attention operation is calculated for the features between different frames. Specifically, each feature map in the feature is divided into non-overlapping cubes along the height and width dimensions, and multi-head self-attention operations are performed in these cube areas. By stacking multiple temporal Transformer modules, the feature expression capability of the temporal dimension can be gradually enhanced while maintaining the integrity of the spatial information.

[0061] After all the time Transformer modules, the third feature is obtained ,Finally, deconvolution is constructed to perform upsampling operations to reconstruct the video frame and obtain the final restoration result.

[0062] The test platform, test data, test process and test results used in this embodiment are as follows.

[0063] Test platform: The experimental tests were conducted on a computer equipped with an Intel Core i9-13900K and an NVIDIA GeForce RTX 4090 GPU. The operating system was Ubuntu 20.04 and implemented using Python 3.10 and PyTorch 2.10.

[0064] Test data: This method is tested on the YouTube-VOS and DAVIS datasets, and an additional custom large-area defect dataset is constructed, with the mask ratio controlled at 40%~50%. The test video resolution is 480p and the frame rate is 30fps. It mainly contains natural scenes, urban street scenes, and human activities. The defect area is constructed by random occlusion and manual annotation to simulate large-area defects in real application scenarios.

[0065] Testing process: The defective video is input into this method for restoration, and compared with the current mainstream restoration method, LPIPS and VFID are used as evaluation indicators to quantify the restoration effect. In addition, a preset number of visual evaluators are invited to subjectively score the rationality, structural integrity and fluency of the restoration content.

[0066] Test results: As shown in Figure 1, the objective indicators of various restoration methods for video restoration with large area defects (mask ratio is 40%~50%) are compared. From the experimental results in Table 1, it can be seen that in terms of objective indicators, the VFID (Video Fréchet Inception Distance, which can effectively evaluate the temporal consistency and dynamic quality of the video. The lower the VFID value, the higher the quality of the generated video) and LPIPS (Learned Perceptual Image Patch Similarity, which can learn image block perceived similarity. LPIPS focuses on the high-level features of the image rather than the differences at the pixel level. The smaller the value, the more similar the images are) indicators of this method are significantly better than other methods.

[0067] Table 1 Comparison of objective indicators of various restoration methods for large-area defective video restoration;

[0068]

[0069] like Figure 5 As shown, the present invention also discloses a large-area defective video restoration device based on frequency domain fusion, comprising:

[0070] A downsampling module 501 is used for the downsampling module, which is used to obtain a defective video frame sequence and downsample a plurality of continuous video frames in the video frame sequence to obtain a corresponding first feature;

[0071] The frequency domain fusion residual processing module 502 is used to input the first feature into the stacked frequency domain fusion residual block, and use the stacked frequency domain fusion residual block to perform global information modeling on the first feature; each layer of the frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence; each frequency domain cross fusion module receives the input feature, and decomposes the input feature along the channel dimension into local features and global features based on the channel ratio; the local feature is input into the local branch, and the local branch uses 3×3 ordinary convolution operation to extract local spatial features from the local feature; the global feature is input into the global branch, and the global branch uses adaptive frequency domain convolution filtering The detector module converts the global features from the spatial domain to the frequency domain, strengthens and learns the global features, fully captures the global context information, and then restores it to the spatial domain; the local branch and the global branch achieve information complementarity through the cross-fusion mechanism, and splice in the channel dimension to generate output features; wherein the two frequency domain cross-fusion modules are respectively the first frequency domain cross-fusion module and the second frequency domain cross-fusion module, the input feature of the first frequency domain cross-fusion module is the first feature, the input feature of the second frequency domain cross-fusion module is the output feature of the first frequency domain cross-fusion module, and the output feature of the second frequency domain cross-fusion module is the second feature;

[0072] A time consistency processing module 503 is used to input the second feature into a stacked time Transformer module, optimize the time consistency between frames through the stacked time Transformer module, and obtain a third feature;

[0073] The up-sampling module 504 is used to up-sample the third feature to reconstruct the video frame to obtain a final restored video.

[0074] The specific implementation of each module of a large-area defective video repair device based on frequency domain fusion and the large-area defective video repair method based on frequency domain fusion are not repeated in this embodiment.

[0075] Figure 6 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, the electronic device of this embodiment includes: a processor 601 and a memory 602; wherein the memory 602 is used to store computer-executable instructions; the processor 601 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0076] Optionally, the memory 602 may be independent or integrated with the processor 601 .

[0077] When the memory 602 is independently provided, the electronic device further includes a bus 603 for connecting the memory 602 and the processor 601 .

[0078] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 601 executes the computer execution instructions, the above method is implemented.

[0079] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 601, the above method is implemented.

[0080] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0081] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0082] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0083] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 601 to perform some steps of the methods of various embodiments of the present application.

[0084] It should be understood that the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 601 may be any conventional processor 601, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 601, or may be executed by a combination of hardware and software modules in the processor 601.

[0085] The memory 602 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0086] The bus 603 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 603 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 603 in the drawings of the present application is not limited to only one bus 603 or one type of bus 603.

[0087] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0088] An exemplary storage medium is coupled to the processor 601, so that the processor 601 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 601. The processor 601 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 601 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0089] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for restoring large-area defective video based on frequency domain fusion, characterized in that: include: Acquire a defective video frame sequence, and downsample a plurality of continuous video frames in the video frame sequence to obtain a corresponding first feature; The first feature is input into the stacked frequency domain fusion residual block, and the stacked frequency domain fusion residual block is used to perform global information modeling on the first feature; each layer of the frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence; each frequency domain cross fusion module receives the input feature, and decomposes the input feature into local features and global features along the channel dimension based on the channel ratio; the local feature is input into the local branch, and the local branch uses a 3×3 ordinary convolution operation on the local feature to extract the local spatial feature; the global feature is input into the global branch, and the global branch converts the global feature from the spatial domain to the frequency domain through an adaptive frequency domain convolution filter module, strengthens and learns the global feature, fully captures the global context information, and then restores it to the spatial domain; the local branch and the global branch complement each other through the cross fusion mechanism, and splice in the channel dimension to generate the output feature; wherein the two frequency domain cross fusion modules are respectively the first frequency domain cross fusion module and the second frequency domain cross fusion module, the input feature of the first frequency domain cross fusion module is the first feature, the input feature of the second frequency domain cross fusion module is the output feature of the first frequency domain cross fusion module, and the output feature of the second frequency domain cross fusion module is the second feature; The second feature is input into the stacked temporal Transformer module, and the temporal consistency between frames is optimized through the stacked temporal Transformer module to obtain the third feature; The third feature is upsampled to reconstruct the video frame and obtain the final repaired video.

2. The method for repairing large-area defective video based on frequency domain fusion according to claim 1, characterized in that: Each frequency domain cross-fusion module receives input features and decomposes the input features into local features and global features along the channel dimension based on the channel ratio, including: Each frequency domain cross fusion module receives input features ;in, , and Represent the height, width and number of channels of the feature respectively, represents the set of real numbers; Input features Decompose into local features along the channel dimension and global features ;in, Indicates the ratio of channels allocated to global features.

3. The method for repairing large-area defective video based on frequency domain fusion according to claim 2, characterized in that: The processing and cross-fusion process of local branches and global branches are as follows: ； ； ； in, Represents the output features of the local branch; represents the activation function; represents normalization; Represents a 3×3 normal convolution operation; Represents the output features of the global branch; Represents normalization and activation function; represents an adaptive frequency domain convolution filter that converts features into the frequency domain to capture global context information; represents the output features; Indicates that and Perform concatenation on the channel dimension.

4. The method for repairing large-area defective video based on frequency domain fusion according to claim 3 is characterized in that: The processing process of the adaptive frequency domain convolution filter is as follows: Adaptive frequency domain convolution filter receives global features First, the fast Fourier transform (FFT) is used to convert the global features into the corresponding frequency domain representation, and the imaginary part and the real part of the complex result are concatenated to obtain the frequency domain features. , and then the frequency domain features are adjusted by adaptively learning the filter weights Update the global information, expressed as: ； in, Represents the frequency domain features after global information update; represents element-wise product; Represents the adaptive filter weights learned from frequency domain features.

5. The method for repairing large-area defective video based on frequency domain fusion according to claim 4 is characterized in that: The learning process of adaptive filter weights is as follows: First, a 1×1 group convolution is used for feature mapping, then the ReLU activation function is applied, and finally another 1×1 group convolution is used to generate the filter weights so that the frequency domain features can complete the global information update; the updated frequency domain features Split into real and imaginary parts along the channel dimension, reconvert back to the representation of the integer, and finally convert back to the spatial domain features through the inverse fast Fourier transform IFFT .

6. The method for restoring large-area defective video based on frequency domain fusion according to claim 1, characterized in that: The processing of the temporal Transformer module is as follows: The temporal Transformer module receives the second feature processed by the frequency domain fusion residual block, divides each feature map in the second feature into non-overlapping cubes along the height and width dimensions, performs multi-head self-attention operations in these cube areas, and obtains the third feature .

7. A large area defective video restoration device based on frequency domain fusion, characterized in that: include: A downsampling module, used to obtain a defective video frame sequence, and downsample a plurality of continuous video frames in the video frame sequence to obtain a corresponding first feature; The frequency domain fusion residual processing module is used to input the first feature into the stacked frequency domain fusion residual blocks, and use the stacked frequency domain fusion residual blocks to perform global information modeling on the first feature; each layer of the frequency domain fusion residual block is composed of two frequency domain cross fusion modules connected in sequence; each frequency domain cross fusion module receives the input feature, and decomposes the input feature into local features and global features along the channel dimension based on the channel ratio; the local feature is input into the local branch, and the local branch uses a 3×3 ordinary convolution operation to extract local spatial features from the local feature; the global feature is input into the global branch, and the global branch uses an adaptive frequency domain convolution filter model The block converts the global features from the spatial domain to the frequency domain, strengthens and learns the global features, fully captures the global context information, and then restores it to the spatial domain; the local branch and the global branch achieve information complementarity through the cross-fusion mechanism, and splice in the channel dimension to generate output features; wherein the two frequency domain cross-fusion modules are respectively the first frequency domain cross-fusion module and the second frequency domain cross-fusion module, the input feature of the first frequency domain cross-fusion module is the first feature, the input feature of the second frequency domain cross-fusion module is the output feature of the first frequency domain cross-fusion module, and the output feature of the second frequency domain cross-fusion module is the second feature; A time consistency processing module, used for inputting the second feature into a stacked time Transformer module, optimizing the time consistency between frames through the stacked time Transformer module, and obtaining a third feature; The upsampling module is used to upsample the third feature to reconstruct the video frame to obtain the final restored video.

Images