AI generation video detection method based on time sequence difference and wavelet feature mapping

Abstract

The invention relates to an AI generated video detection method based on time sequence difference and wavelet feature mapping. The method mainly comprises the following steps: a U-Net feature extraction module based on a time domain center difference operator effectively inhibits static redundant features by enhancing the perception capability of inter-frame tiny dynamic changes, and focuses on extracting high-frequency discrimination information from continuous video frames; and a Transform feature mapping module based on wavelet attention guides attention distribution in a wavelet domain, highlights the structural and global features of the video, and realizes accurate feature representation of low and medium frequency regions. The two modules cooperate to form a complementary feature representation mechanism through accurate extraction of high-frequency dynamic information and effective modeling of low and medium-frequency structural semantics under the background of rapid evolution of multiple generative models, so that the overall detection precision and the adaptive capacity of the model are remarkably improved, and the detection efficiency is improved. And a technical support is provided for constructing an efficient AI generation video detection system.

Description

Technical Field

[0001] The present invention relates to the field of AI-generated video detection, and in particular to an AI-generated video detection method based on temporal difference and wavelet feature mapping. Background Art

[0002] AI-generated video detection is a key digital content analysis technology used to identify AI-generated video content and maintain the credibility and authenticity of digital information. With the rapid development of generative AI, the application of techniques such as deepfakes, generative adversarial networks, and diffusion models has become increasingly widespread, giving rise to AI-generated video detection technology. This technology uses methods such as pixel-level detail analysis, spatiotemporal consistency detection, and metadata feature extraction to comprehensively analyze video content, accurately identifying traces of generated content and protecting the digital content ecosystem from the spread of fabricated information, privacy abuse, and social trust crises. AI-generated video detection technology has broad application potential, encompassing copyright protection, judicial evidence collection, news dissemination, and social media, providing technical support for preventing digital fraud and building a trustworthy digital society. However, the current diversity and rapid iteration of AI-generated models pose significant challenges to detection technology. The constant updating of generative models requires detection algorithms to continuously adapt to new generation patterns, creating a dynamic competitive environment. Therefore, continuous optimization of AI-generated video detection technology is crucial. By incorporating innovative methods such as deep learning and feature separation, this technology is expected to achieve higher-precision recognition capabilities in various scenarios, providing a reliable solution for digital content authenticity verification and copyright protection, and ensuring a balanced development between technological advancement and social ethics.

[0003] Traditional video generation techniques often focus on the human face, and detection techniques also primarily focus on verifying the authenticity of facial images. "Hanyi Wang, Zihan Liu, and Shilin Wang, "Exploiting Complementary Dynamic Incoherence for DeepFake Video Detection," Transactions on Circuits and Systems for Video Technology, 2023, pp. 4027-4040," analyze the dynamic features of the full face and the local mouth region in parallel, achieving bidirectional reinforcement of local and global information at multiple scales to distinguish video authenticity. "Jongwook Choi, Taehoon Kim, Yonghyun Yeong, Seungryul Baek, and Jongwon Choi, "Exploiting Style Latent Flows for Generalizing Deepfake Video Detection," Conference on Computer Vision and Pattern Recognition, 2024, pp. 1133-1143," propose a video detection method based on dynamic modeling of style latent variables. This method improves detection accuracy and generalization by integrating a temporal style encoding unit with a style attention mechanism. With the rapid expansion of non-face video generation technology, video detection technology for non-face generated content has also gradually developed. "Peisong He, Leyao Zhu, Jiaxing Li, Shiqi Wang, and Haoliang Li, "Exposing AI-generated Videos: ABenchmark Dataset and a Local-and-Global Temporal Defect Based Detection Method," arXiv preprint, 2024, arXiv:2405.04133," exploits the temporal characteristics of generated videos by integrating local motion information with global appearance changes. This paper explores the difference between the high temporal redundancy between real video frames and the lack of coherence between generated video frames.With the widespread application of diffusion models, "QingyuanLiu, Pengyuan Shi, Yun-Yun Tsai, Chengzhi Mao, Junfeng Yang, "Turns Out I'm NotReal: Towards Robust Detection of AI-Generated Videos," arXiv preprint, 2024, arXiv: 2406.09601." Based on CNN and LSTM, a new AI-generated video detection framework called DIVID is proposed. It comprehensively utilizes spatial and temporal features and combines the diffusion reconstruction error DIRE with RGB frame analysis to improve the accuracy of the detection algorithm both inside and outside the domain. Regarding the content expression of the video, "Chirui Chang, Zhengzhe Liu, Xiaoyang Lyu, Xiaojuan Qi, "What Matters in Detecting AI-Generated Videos like Sora? ”arXiv preprint, 2024, arXiv:2406.19568.” Through three 3D convolutional networks, the differences between real videos and AI-generated videos in appearance, motion, and geometry are revealed, further proving that there are still significant differences between the two in physical logic. To further utilize the continuous characteristics of videos, “Haoxing Chen, Yan Hong, Zizheng Huang, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Jun Lan, Huijia Zhu, Jianfu Zhang, Weiqiang Wang, Huaxiong Li, “DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmark,” arXiv preprint, 2024, arXiv:2405.19707.” proposed a plug-and-play DeMamba module to identify AI-generated videos by analyzing the inconsistencies in the temporal and spatial dimensions of videos. Although existing AI-generated video detection methods can be used to a certain extent, with the continuous evolution and updating of the generation model structure, the visual quality and spatiotemporal consistency of the synthesized videos are becoming increasingly realistic, posing a challenge to previous detection methods. Therefore, detection technology urgently needs to improve its adaptability to diverse generation modes, thereby enhancing the generalization of the model to meet the requirements of versatility and stability in practical application scenarios. To solve the above problems, the present invention proposes an AI-generated video detection method based on temporal difference and wavelet feature mapping. Summary of the Invention

[0004] The present invention proposes an AI-generated video detection method based on temporal difference and wavelet feature mapping to achieve high-accuracy detection of videos generated by mainstream models. The method mainly includes: proposing a feature extraction method based on time domain central difference and a feature mapping method based on wavelet attention.

[0005] The specific contents are as follows:

[0006] (1) A feature extraction method based on temporal central difference is proposed: the central difference operator is introduced into the convolution operation to construct a new U-net network, which improves the ability to perceive and extract features of changes between video frames. Compared with the traditional U-net network, the constructed network can effectively suppress the extraction of static redundant features, significantly improve the network's sensitivity to temporal information, and accurately capture and efficiently utilize the key features of high-frequency information.

[0007] The overall AI-generated video detection method adopts a dual-stream framework, which mainly includes four parts: secondary wavelet decomposition and reconstruction, high-frequency time domain central difference U-net network, medium and low-frequency wavelet attention Transformer network and classifier. Figure 1 As shown in the figure, the specific execution process of the video detection framework is as follows: the original video V is first decomposed into high-frequency components hh by the first-level wavelet transform 1 ,hl 1 、lh 1 and low-frequency components ll 1 . Low frequency component ll 1 Further decomposed into intermediate frequency components hh by the second level wavelet transform 2 ,hl 2 、lh 2 and low-frequency components ll 2 By inverse wavelet transform, high frequency, medium frequency and low frequency components are reconstructed respectively to obtain high frequency reconstruction V h , intermediate frequency reconstruction V m and low-frequency reconstruction V l . High frequency reconstruction V h Input to the time domain central difference U-net network to extract the texture, edge, and inter-frame change details of the video, and output the high-frequency feature vector L through linear embedding operation h . IF reconstruction V m and low-frequency reconstruction V l Input to the wavelet attention Transformer network to capture global features and generate medium and low frequency feature vectors L ml . High frequency eigenvector L h and the low- and medium-frequency eigenvectors L ml The feature fusion module performs splicing to generate a comprehensive feature vector. Finally, the comprehensive feature vector is input into the classifier for authenticity judgment.

[0008] The temporal center difference feature extraction method in this invention relies on the U-net network, and its input is a tensor with 8 frames, 3 channels, and a pixel size of 224×224. The tensor is generated as follows: 8 consecutive RGB frames are randomly extracted from the original video V, and a preprocessed tensor is generated by center cropping and data enhancement. Then, the input V is obtained by the first wavelet transform decomposition and high-frequency reconstruction. h The time-domain central difference U-net network generates high-frequency features of the same size as the input tensor. The feature is linearly embedded to obtain a vector of dimension 512 and number n to align the mapping features of the other stream.

[0009] Traditional 3D convolutions focus too much on local pixel intensity, lacking the ability to sensitively perceive dynamic differences between frames and failing to explicitly model the changes themselves. Temporal central difference convolution is a lightweight improvement that enhances the perception of inter-frame differences by introducing a central difference operator. This introduces the idea of ​​central difference, allowing the model to focus more on the changing trend of the input rather than the absolute value itself. Given a standard 3D convolution operation:

[0010] f conv (x)=w*x (1)

[0011] Where x is the input and w is the convolution kernel. In the central difference convolution, add a central difference operator:

[0012] f cdc (x) = w*x + θ*D central (x) (2)

[0013] Among them, θ is a learnable weight coefficient used to adjust the ratio, D central (x) is the central difference operator, specifically:

[0014] D central (x) = x t+Δt -x t-Δt (3)

[0015] like Figure 2As shown, in the present invention, the time-domain central difference U-net network is mainly composed of upsampling and downsampling modules. The downsampling module first uses a 3D convolution layer to perform preliminary feature extraction on the input video, with a convolution kernel size of 3×5×5 and the number of output channels being 1 / 4 of the number of input channels. Subsequently, batch normalization and ReLU activation function are used to enhance the feature expression capability, and downsampling is performed in the spatial dimension through a 3D maximum pooling layer. Then, dynamic features are extracted through two time-domain central difference convolutions with a convolution kernel size of 3. Each layer is followed by batch normalization, ReLU activation function and 3D maximum pooling layer, which gradually reduce the spatial resolution while increasing the number of channels. The upsampling module is mainly composed of three upsampling layers, each of which amplifies the feature map in the spatial dimension through bilinear interpolation, and then a 3D convolution layer with a convolution kernel size of 3×1×1 is used to adjust the number of channels, and batch normalization and ELU activation function are used to enhance the feature expression capability. Finally, a convolution layer is used to map the upsampled feature map to the same number of channels as the input video to obtain high-frequency key features.

[0016] (2) A feature mapping method based on wavelet attention is proposed: With the wavelet attention mechanism as the core, a new multi-level Transformer network is designed and constructed. By introducing wavelet transform to decompose the input features in the frequency domain, the effective modeling and highlighting of medium and low frequency features are achieved. Compared with the traditional Transformer network that only relies on spatial self-attention to model long-range dependencies, the present invention uses the characteristics of wavelet domain decoupling to enable the network to have stronger global perception and local feature response capabilities in the frequency space, thereby alleviating the problem that traditional self-attention is easily disturbed in complex backgrounds and lacks sensitivity to fine-grained discriminant features. Overall, the proposed wavelet attention Transformer structure, while maintaining the advantages of long-range dependency modeling, further improves the analysis and discrimination capabilities of medium and low frequency deep structural information, significantly enhancing the stability and generalization performance of the model in complex generation scenarios.

[0017] The wavelet attention feature mapping method in this invention relies on the Transformer network, and its input is a tensor with 8 frames, 3 channels, and a pixel size of 224×224. The tensor is generated as follows: 8 consecutive RGB frames are randomly extracted from the original video V, and a preprocessed tensor is generated by center cropping and data enhancement. Then, the input V is decomposed and reconstructed by the first and second wavelet transforms and high-frequency reconstruction. m and V l The wavelet attention transformer network processes the input in multiple layers and finally generates a vector of dimension 512 and number n.

[0018] The wavelet attention Transformer network consists of 8 Transformer blocks, each of which is mainly composed of wavelet attention, spatial attention, temporal attention modules and feedforward network. Figure 3 As shown, the first Transformer block reconstructs the intermediate frequency V m and low-frequency reconstruction V l As input, it is first processed by the video frame block encoding module. This module decomposes the video into multiple blocks, and each block is normalized after encoding. The block sequence is decomposed into sub-bands of different frequencies through wavelet transform, and the attention mechanism is combined to capture the differences in local frequency domain features, thereby enhancing the modeling ability of medium and low frequency features. Spatial attention captures important areas in the feature map in the spatial dimension and suppresses the interference of background redundant information. Temporal attention models long-range dependencies in the temporal dimension to enhance the perception of dynamic changes. The output of the attention mechanism is normalized and processed by the feedforward network, expanded and divided into two features, and then input into the next Transformer block. After the eighth and last Transformer block, a feature linear mapping operation is added to output effective medium and low frequency key features. These feature vectors are fused with the high-frequency feature vectors and finally input into a classifier that alternately stacks linear layers and nonlinear activations (such as Figure 5 (as shown) to make a judgment.

[0019] In each Transformer block, the specific design of wavelet attention is as follows: Figure 4 As shown, the intermediate frequency reconstruction V m and low-frequency reconstruction V l After block coding and normalization, the query branch q and key branch k are generated by convolution operations with kernels of 1×1 and 3×3 and wavelet transform respectively. l After block coding and normalization, the value branch v is generated by convolution operations with kernels of 1×1 and 3×3. The features after multiplication of query branch q and key branch k are returned to the spatial domain through inverse wavelet transformation to obtain feature T:

[0020] T=W -1 (q*k) (4)

[0021] Among them, W -1 For the inverse wavelet transform, T is normalized and multiplied by the value branch v to obtain the feature T att :

[0022] T att =N(T)*v (5)

[0023] Among them, N(·) is the normalization operation, T attAfter the convolution operation with a convolution kernel of 1×1 and the sum branch v is added, the feature X perceived by the wavelet attention is finally obtained:

[0024] X=v+f conv1×1 (T att ) (6)

[0025] Among them, f conv1×1 (·) is a convolution operation with a kernel of 1×1. The wavelet attention mechanism can effectively extract the deep structural information of low- and medium-frequency features by combining wavelet transform and attention mechanism.

[0026] Compared with existing technologies, the above method has at least the following significant effects:

[0027] 1. The present invention proposes a feature extraction method based on time-domain central difference, which is used to efficiently extract time-sensitive high-frequency discriminant features from a continuous video frame sequence. This method designs a U-Net structure that introduces a central difference operator. By enhancing the ability to respond to small dynamic changes between frames, it effectively suppresses redundant static information and highlights high-frequency change areas. The encoding path of the network adopts a layer-by-layer downsampling strategy to obtain multi-scale features, and the decoding path is gradually reconstructed through a cross-layer feature fusion mechanism. While maintaining temporal continuity, it significantly improves the expression quality and discrimination performance of high-frequency semantic features, providing more stable and higher-precision feature support for subsequent AI-generated content recognition tasks.

[0028] 2. The present invention proposes a feature mapping method based on wavelet attention to enhance the model's ability to model the structural and global distribution features in AI-generated content. This method constructs a Transformer network architecture that integrates discrete wavelet transform and attention mechanism, focusing on the characterization of medium and low-frequency feature components. By performing attention-guided modeling in the wavelet domain, the network can effectively suppress background noise and local interference while retaining the global contextual semantic information of the image, thereby achieving accurate extraction of medium and low-frequency areas with discriminative significance. This method has good generalization ability and is suitable for AI-generated video detection scenarios under complex generation modes, especially in the context of rapid evolution of diversified generation models, showing stronger adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the framework of the AI-generated video detection method of the present invention;

[0030] Figure 2 Schematic diagram of the novel U-net network framework based on time-domain central difference of the present invention;

[0031] Figure 3Schematic diagram of the novel Transformer network framework based on wavelet attention in the present invention;

[0032] Figure 4 Schematic diagram of the wavelet attention mechanism structure of the present invention;

[0033] Figure 5 Schematic diagram of the feature classifier structure of the present invention. DETAILED DESCRIPTION

[0034] The present invention is an AI-generated video detection method based on temporal difference and wavelet feature mapping. The following describes specific implementations of the present invention using mainstream operations in the same field as examples. Those skilled in the art should understand that these implementations are merely intended to illustrate the technical principles of the present invention and are not intended to limit the scope of application of the present invention.

[0035] The development language is Python, and the deep learning framework is Pytorch. The specific steps are as follows:

[0036] Step 1: Data selection. The data for this study consists of two parts, totaling 70,000 video samples. Among them, 35,000 video samples come from the negative samples of the GenVideo dataset, which are marked as 0-Fake after processing, representing AI-generated video data; the other 35,000 video samples come from the positive samples of the GenVideo dataset, marked as 1-Real, representing real-shot video data. The generative models for AI-generated video data include Crafter, Gen2, HotShot, Lavie, ModelScope, MoonValley, MorphStudio, Show 1, Sora, and WildScrape, ensuring the diversity and representativeness of the data. The real-shot video data mainly comes from the Youku video platform and is strictly screened to ensure its authenticity. All data are divided into training set, validation set, and test set in a ratio of 10:1:1, which are used for model training, hyperparameter adjustment, and final performance evaluation, respectively.

[0037] Step 2: Data Augmentation. Before data loading, the video frames are normalized and augmented. First, the video frames are resized to 256×256 via center cropping and then uniformly scaled to 224×224 to meet the input requirements of the method. The augmentation strategy is selected based on probability and includes random horizontal flips with a probability of 50%, random image compression with a probability of 50% and a quality range of [50, 100], Gaussian noise with a probability of 30%, Gaussian blur with a kernel size of [3, 5] with a probability of 30%, and grayscale conversion with a probability of 0.1%. These augmentation operations can be performed independently or in combination. After completion, the video frames are uniformly normalized using the mean and standard deviation of the ImageNet dataset. This operation constructs a training set of samples with certain transformation perturbations, providing standardized and diverse input data for subsequent model training.

[0038] Step 3: Build an end-to-end AI-generated video detection framework. The original video is first subjected to a two-level discrete wavelet transform to extract multi-scale high- and low-frequency components. Feature modeling is then performed on the high- and low-frequency paths separately. The high-frequency path is reconstructed using inverse wavelet transforms and then fed into a time-domain central difference U-Net to extract temporal variations. The reconstructed low-frequency path is then fed into a wavelet attention transformer to capture global semantics and temporal dependencies. Finally, the high-frequency features are concatenated and fused with the mid- and low-frequency features, and then fed into a classifier to verify video authenticity. The overall process integrates frequency domain decomposition, temporal modeling, and multi-frequency feature fusion to form a unified framework for video forgery detection.

[0039] Step 4: Model Training. During the training phase, a binary cross-entropy loss function with logits was used as the loss function to improve the model's stability and numerical efficiency in binary classification generation and detection tasks. The Adam optimizer with an initial learning rate of 0.00001 was used to control the gradient update process, and a weight decay coefficient of 0.00001 was introduced to enhance the model's generalization ability. A milestone-based multi-step decay strategy was used for the learning rate schedule, multiplying the learning rate by a decay factor of 0.1 after the 20th and 25th batches. Regarding the optimization strategy, the total number of training epochs was set to 30, with each epoch consisting of 8,000 training batches. During the validation phase, 800 batches were sampled per epoch, with a batch size of 4 for both training and validation. After each training batch, the validation set was used to perform inference evaluation on the current model, guiding hyperparameter tuning during training to monitor overfitting risks and facilitate optimal model selection. The entire training process achieved end-to-end optimization under this configuration.

[0040] Step 5: Testing. The trained detection model was fully evaluated on an independent test set, focusing on overall discrimination performance and generalization ability. During the test, the model weights used the best parameters saved when the verification accuracy was the highest during the training process, and forward reasoning was performed in the Eval mode without enabling gradient updates to ensure the stability and fairness of the test process. Each test sample was processed according to the unified processing method in the training phase, and high-frequency features L were extracted separately. h With the mid-low frequency characteristics L ml After end-to-end inference, the model outputs an AI-generated probability score. Finally, the category is determined based on a set threshold, which is then compared with the label to calculate multiple metrics. For performance evaluation, accuracy (ACC), average precision (AP), and area under the receiver operating characteristic (ROC) curve (AUC) can be used to measure the model's basic classification capability, robustness, and overall discrimination ability, respectively.

[0041] Step 6: Application. During the application phase, the model performs inference and judgment using the same process as the testing phase, ensuring that input preprocessing, feature extraction, and prediction logic are completely consistent to ensure the accuracy and consistency of the inference results.

[0042] In summary, for scenarios where video generation technology iterates quickly, the present invention designs an AI-generated video detection method based on temporal difference and wavelet attention. This method effectively captures hidden generation features in AI-generated videos by mining high-frequency dynamic changes and medium- and low-frequency structural information in consecutive frames. The design fully considers the feature differences presented by different types of generation models to achieve wide adaptability to multiple generation paradigms. Experiments have shown that the present invention exhibits superior detection performance on multiple mainstream synthetic datasets, has higher recognition accuracy and stronger generalization ability, and can provide more robust and reliable technical support for deep fake identification in practical applications.

[0043] Those skilled in the art will understand that the scope of protection of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for detecting AI-generated videos based on temporal difference and wavelet feature mapping, characterized in that: A time-domain central difference U-net network is constructed to enhance high-frequency expression in the wavelet domain. This network can effectively suppress the extraction of static redundant features, significantly improve the network's sensitivity to temporal information, and accurately capture and efficiently utilize the key features of high-frequency information. Combined with the wavelet attention-based Transformer network, a complementary feature representation mechanism is formed, significantly improving overall detection accuracy and adaptability. The specific contents include: U-net feature extraction network based on time domain central difference, Transformer feature mapping network based on wavelet attention; the first level wavelet transform of the original video V is decomposed into high-frequency components hh 1 ,hl 1 、lh 1 and low-frequency components ll 1 , low frequency component ll 1 Further decomposed into high frequency components hh by the second level wavelet transform 2 ,hl 2 、lh 2 and low-frequency components ll 2 , through inverse wavelet transform, the high frequency, medium frequency and low frequency components are reconstructed respectively to obtain the high frequency reconstruction V h , intermediate frequency reconstruction V m and low-frequency reconstruction V l ; High frequency reconstruction V h Input to the time domain central difference U-net network to extract the texture, edge, and inter-frame change details of the video, and output the high-frequency feature vector L through linear embedding operation h ; IF reconstruction V m and low-frequency reconstruction V l Input to the wavelet attention Transformer network to capture global features and generate medium and low frequency feature vectors L ml ; High-frequency feature vector L h and the low- and medium-frequency eigenvectors L ml The feature fusion module is used to combine and generate a comprehensive feature vector. Finally, the comprehensive feature vector is input into the classifier for authenticity judgment. The U-net feature extraction network based on time-domain central difference includes time-domain central convolution and 3D convolution operations. The downsampling module first uses a 3D convolution layer to perform preliminary feature extraction on the input video, with a convolution kernel size of 3×5×5 and an output channel number of 1 / 4 of the input channel number; then, batch normalization and ReLU activation function are used to enhance the feature expression ability, and downsampling is performed in the spatial dimension through a 3D maximum pooling layer; then, dynamic features are extracted through two time-domain central difference convolutions with a convolution kernel size of 3; each layer is followed by batch normalization, ReLU activation function and 3D maximum pooling layer to gradually reduce the spatial resolution while increasing the number of channels; the upsampling module is mainly composed of three upsampling layers, each layer amplifies the feature map in the spatial dimension through bilinear interpolation, and then uses a 3D convolution layer with a convolution kernel size of 3×1×1 to adjust the number of channels, and enhances the feature expression ability through batch normalization and ELU activation function; finally, a convolution layer is used to map the upsampled feature map to the same number of channels as the input video to obtain high-frequency key features.

2. The AI-generated video detection method based on time-domain central difference and wavelet attention according to claim 1 is characterized in that: The wavelet-based attention-based Transformer network, by modeling attention guidance in the wavelet domain, can effectively suppress background noise and local interference while preserving the global contextual semantic information of the image, achieving accurate mapping of the discriminative low- and mid-frequency regions. Specific contents include: The wavelet attention Transformer network consists of 8 Transformer blocks, each of which is mainly composed of wavelet attention, spatial attention, temporal attention modules and feedforward network; the first Transformer block reconstructs the intermediate frequency V m and low-frequency reconstruction V l As input, it is first processed by the video frame block encoding module; this module decomposes the video into multiple blocks, and each block is normalized after encoding; the block sequence is decomposed into sub-bands of different frequencies through wavelet transform, and the attention mechanism is combined to capture the differences in local frequency domain features, enhancing the modeling ability of medium and low frequency features; spatial attention captures important areas in the feature map in the spatial dimension and suppresses the interference of background redundant information; temporal attention models long-range dependencies in the temporal dimension and improves the perception of dynamic changes; the output of the attention mechanism is normalized and processed by the feedforward network, expanded and divided into two features, and then input into the next Transformer block; after the eighth and last Transformer block, a feature linear mapping operation is added to output effective medium and low frequency key features; these feature vectors are fused with the high-frequency feature vectors and finally input into a classifier with alternating stacked linear layers and nonlinear activations for judgment; In each Transformer block, the specific design of wavelet attention is as follows: intermediate frequency reconstruction V m and low-frequency reconstruction V l After block coding and normalization, the query branch q and key branch k are generated by convolution operations with convolution kernels of 1×1 and 3×3 and wavelet transform respectively; low-frequency reconstruction V l After block coding and normalization, the value branch v is generated by convolution operations with convolution kernels of 1×1 and 3×3; the features after multiplication of the query branch q and the key branch k are returned to the spatial domain through inverse wavelet transformation to obtain the feature T: T = W -1 (q*k), where W -1 For the inverse wavelet transform, T is normalized and multiplied by the value branch v to obtain the feature T att :T att =N(T)*v, where N(·) is the normalization operation, T att After the convolution operation with a convolution kernel of 1×1, the sum branch v is added and the feature X perceived by the wavelet attention is finally obtained: X=v+f conv1×1 (T att ), where f conv1×1 (·) is a convolution operation with a convolution kernel of 1×1. The wavelet attention mechanism can effectively extract the deep structural information of medium and low frequency features by combining wavelet transform and attention mechanism.

Images