A video anomaly detection method based on MSGRU-TRF

Abstract

A kind of video anomaly detection method based on MSGRU-TRF relates to computer vision and pattern recognition field.The present application proposes new MSGRU-TRF network for video, including three modules.Based on the multi-scale time feature learning module of GRU, GRU structure is added in multi-scale convolution network, both the characteristics of multi-scale convolution network are utilized to the multi-scale modeling of input data, and the modeling on the time of input data is strengthened.The time resolution feature mapping module of the present application maps the time resolution information of video feature to the feature learning network auxiliary network to learn more information of video.The present application proposes adaptive k module to improve the traditional Top-k loss function, by customizing the value of k for different input videos, more flexibility is brought for network training.The present application has good time scale modeling effect on real world long and short monitoring videos, and has higher accuracy in video anomaly detection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition, and relates to a video anomaly detection method for MSGRU-TRF networks. Background Technology

[0002] Social security is a fundamental condition for ensuring the continuous development of a nation's comprehensive strength. Only sound security measures can protect and develop social productivity and promote the sustained and healthy development of society and the economy. Surveillance video plays a crucial role in maintaining social security. However, currently, the main functions of surveillance video systems are displaying surveillance footage and storing video data; anomaly detection relies on security personnel watching monitors. Anomalies occur infrequently and are unpredictable, making manual detection prone to false positives and false negatives. Furthermore, timely detection of anomalies from massive amounts of video data requires significant human resources. While the cost of installing surveillance equipment is decreasing, the manpower required to monitor and analyze this video data remains very expensive. Therefore, relying on manual anomaly detection is uneconomical for businesses. Thus, building an intelligent video anomaly detection system that continuously analyzes and monitors video surveillance data 24 / 7, automatically detecting and issuing warnings for ongoing anomalies, is both meaningful and necessary.

[0003] With the development and application of deep learning in computer vision, using deep learning to solve abnormal behavior detection in surveillance videos has become one of the research hotspots in recent years. Because video anomalies are characterized by their small number and difficulty in definition, existing methods mainly use multi-event learning-based approaches within weakly supervised methods. To address the strong spatiotemporal correlation of video data, some algorithms have added multi-scale convolutional structures to the feature extraction network within the multi-event learning framework to enhance the algorithm's temporal modeling capabilities. However, this convolution-based multi-scale structure often fails to effectively extract global temporal relationships. Summary of the Invention

[0004] To address the shortcomings of existing algorithms, this invention proposes a high-precision video anomaly detection algorithm based on the MSGRU-TRF network. A GRU structure is incorporated into a multi-scale convolutional network, leveraging the multi-scale modeling capabilities of the input data while further enhancing the temporal modeling. Furthermore, this invention extracts temporal resolution information from different video features present in the preprocessing stage of common multi-event learning methods, mapping this information to a feature learning network via a temporal resolution feature mapping module to assist the network in learning more video information. Finally, this invention proposes an adaptive k-module to improve the traditional Top-k loss function. By customizing the value of k for different input videos, the adaptive k-module brings greater flexibility to network training. Compared to existing algorithms, the proposed MSGRU-TRF network demonstrates excellent temporal scale modeling capabilities for real-world surveillance videos of varying lengths, achieving high accuracy in video anomaly detection.

[0005] This invention provides a high-precision video anomaly detection algorithm based on an MSGRU-TRF network, comprising the following steps:

[0006] 1. Pretreatment

[0007] First, the original long video data is segmented into single video frames. Second, these segments are further divided into 32 fixed segments. Weakly supervised annotations are applied to each video segment at the video level. Finally, a pre-trained I3D feature extractor is used to initially extract motion features from the video, obtaining the feature vectors required for network input. Additionally, the temporal resolution information of the video is calculated as extra input to the network.

[0008] 2. Constructing an MSGRU-TRF network

[0009] The network structure constructed in this invention is as follows:

[0010] A multi-scale GRU temporal feature learning network is constructed, consisting of three dilated convolutional layers (DC1, DC2, DC3), one convolutional layer (C1), three GRU layers (GRU1, GRU2, GRU3), a first feature fusion (F1), a first GRU layer (GRU4), a second feature fusion (F2), a first GRU layer (GRU5), two fully connected layers (FC1, FC2), a third feature fusion (F3), and a module output layer. A temporal resolution feature mapping module consists of two fully connected layers (FC5, FC6). An adaptive k-module consists of two fully connected layers (FC1, FC2), a fourth feature aggregation (F4), two fully connected layers (FC3, FC4), and an adaptive k-module output layer. Finally, a network based on the combination of multi-scale temporal information and temporal resolution information from GRU, known as MSGRU-TRF, is formed.

[0011] 3. Implement an unsupervised video anomaly detection algorithm using the MSGRU-TRF network.

[0012] Training process:

[0013] The original long videos in the training set are segmented into single video frames. These segments are then divided into 32 fixed fragments, and each fragment is labeled with weak supervision at the fragment level. A pre-trained I3D feature extractor is used to initially extract motion features from the videos, obtaining the feature vectors required for network input. Additionally, the temporal resolution information of the videos is calculated as extra input to the network.

[0014] The preprocessed video features and temporal resolution features are input into the multi-scale GRU temporal feature learning module and temporal resolution feature mapping module of the MSGRU-TRF network, respectively. After passing through the multi-scale GRU temporal feature learning module, the video features yield a score matrix containing the anomaly scores for each segment. The temporal resolution information, after passing through the temporal resolution feature mapping module and the adaptive k module, yields a k-value for calculating the Top-k loss function and a feature vector for calculating the Adaptive-k loss function. Network optimization is performed by minimizing the difference between the mean of the k highest-scoring anomaly segments of the input anomaly video and the maximum anomaly score of the normal video. The adaptive k module, with its customized k-value based on the video's specific temporal resolution information, assists in loss function optimization. Finally, the MSE loss function is calculated using the output features of the adaptive k module and preset features to limit the generation of extreme k-values.

[0015] Testing process:

[0016] The original long videos in the test set are segmented into single video frames. A pre-trained I3D feature extractor is used to initially extract motion features from the videos, obtaining the feature vectors required for network input. These feature vectors are then directly input into a trained MSGRU-TRF network to obtain segment-level anomaly scores. The anomaly score for each frame within a segment is the same as the segment's anomaly score. The overall anomaly detection accuracy (AUC) can be calculated using the video anomaly scores and groundtruth anomaly annotations. The difference between the anomaly score and a set threshold is used to determine whether the input video frame is an anomaly.

[0017] Beneficial effects

[0018] 1. This invention proposes a time-resolution feature mapping module.

[0019] Because real-world surveillance videos vary greatly in length, existing weakly supervised video anomaly detection algorithms based on multi-event learning exhibit significant differences in the temporal compression of video features during preprocessing, meaning that the temporal resolution of feature vectors from different videos varies considerably.

[0020] To improve the network's anomaly detection capability for such complex data, this invention proposes a temporal resolution feature mapping module. By utilizing the temporal resolution information calculated in the preprocessing stage, this module maps the additional information unique to each video into a temporal multi-scale feature learning space through a fully connected layer and feature fusion operation. This helps the network learn the temporal resolution differences of video features and improves the detection accuracy of datasets with significantly different video lengths.

[0021] 2. This invention proposes a multi-scale temporal feature learning module based on GRU.

[0022] Current methods utilize multi-scale convolutional networks for learning feature temporal relationships. However, convolutional structures focus more on local temporal dependencies, neglecting the global temporal correlation of the input video data along the timeline, and lacking correlation between branches at different time scales. This invention proposes a multi-scale temporal feature learning module based on GRU. This method combines multi-scale convolution with a GRU structure, leveraging GRU's advantage of better capturing global temporal relationships while overcoming its inability to obtain multi-scale temporal information. Furthermore, this module incorporates a progressive feature fusion structure between different branches, increasing the connections between different time scales and reducing mutual interference. Compared to multi-scale convolution, multi-scale GRU utilizes more global temporal feature information for anomaly learning and also exhibits stronger temporal feature extraction capabilities.

[0023] 3. This invention utilizes the MSGRU-TRF network to implement a weakly supervised video anomaly detection algorithm based on multi-event learning. Existing algorithms that achieve good video anomaly detection results on complex, multi-scene real-world datasets are all based on weakly supervised methods using multi-event learning. However, to facilitate training, existing multi-event learning methods uniformly process input videos of different lengths into feature vectors of the same shape during the preprocessing stage, which leads to differences in the temporal resolution of the features. This invention proposes using the MSGRU-TRF network for video anomaly detection. The three modules in this method (temporal resolution feature mapping module, GRU-based multi-scale temporal feature learning module, and adaptive k module) optimize the problems of multi-event learning from three aspects, further improving the accuracy of video anomaly detection.

[0024] 4. This invention proposes an adaptive k-module.

[0025] Existing weakly supervised methods based on multi-event learning commonly use the Top-k loss function for training. This loss function calculates the difference between the average score of the k highest-scoring segments in an anomalous video and the highest-scoring segment in a normal video. By maximizing this score difference during training, the network can better distinguish between anomalous and normal videos. Current methods all use a pre-defined fixed value k for training, but this reduces training flexibility. The proportion of anomalous frames varies significantly across different anomalous videos; using a fixed value means that the segment scores used to calculate the loss cannot fully represent the anomalous situation of the video, thus limiting the accuracy of anomaly detection.

[0026] This invention proposes an adaptive k-module to optimize the Top-k loss function. By fusing video features with temporal resolution information and reducing dimensionality, this module customizes suitable k values ​​for different videos, increasing training flexibility and improving the training ceiling for anomaly detection. Furthermore, this module proposes an adaptive k-loss function to constrain the generation of extremely large k values. This loss is used as a component of the multi-event learning loss function, thereby further improving anomaly detection accuracy. Attached Figure Description

[0027] Figure 1 The proposed network model structure consists of three parts: a GRU-based multi-scale temporal feature learning module, a temporal resolution feature mapping module, and an adaptive k-module.

[0028] Figure 2 This is the preprocessing flow.

[0029] Figure 3 This is a flowchart of the weakly supervised training process implemented using MSGRU-TRF in this invention.

[0030] Figure 4 This is a flowchart of the algorithm testing process for this invention. Detailed Implementation

[0031] The specific implementation method of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] 1. Pretreatment

[0033] A continuous long video is segmented into individual video frames. These segments are further divided into 32 fixed fragments, and each fragment undergoes video-level weak supervision annotation. The segmented video frames are then input into a pre-trained backbone network, I3D, and the output of the "mix_5c" layer is extracted to obtain the pre-processed video feature vector. The specific network structure is as follows: Figure 2 As shown.

[0034] 2. Construct an MSGRU-TRF network

[0035] like Figure 1 As shown. The specific parameters of each layer of the MSGRU-TRF network constructed in this invention are as follows:

[0036] 1) DC1, DC2, DC3 three-dilated one-dimensional convolutional layers: input size is 2048, input channels are 2048, kernel size is 3, stride is 1, dilation rates are 4, 2, and 1 respectively, edge padding sizes are 4, 2, and 1 respectively, activation function is ReLU, output size is 512, and output channels are 512.

[0037] 2) C1 one-dimensional convolutional layer: input size is 2048, input channels are 2048, kernel size is 1, stride is 1, no edge padding, activation function is ReLU, output size is 512, output channels are 512.

[0038] 3) Three GRU layers: GRU1, GRU2, and GRU3: input size is 512, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, and dropout rate is 0.

[0039] 4) The results from GRU1 and GRU2 are concatenated, with a concatenation dimension of 2.

[0040] 5) F1 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses the ReLU activation function, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024.

[0041] 6) GRU4GRU layer: input size is 1024, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, dropout rate is 0.

[0042] 7) The results obtained through GRU4 are concatenated with the results obtained through GRU3, with a concatenation dimension of 2.

[0043] 8) F2 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses the ReLU activation function, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024.

[0044] 9) GRU5GRU layer: input size is 1024, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, dropout rate is 0.

[0045] 10) The temporal resolution information calculated in the preprocessing stage is copied in terms of feature dimension. The copy dimension is 1, and the size after copying is 32.

[0046] 11) FC1 fully connected layer: input size 1, output size 128, with additional weighting.

[0047] 12) FC2 fully connected layer: input size 128, output size 512, with additional weighting.

[0048] 13) The results from GRU1, GRU4, and GRU5 and the output of FC2 are concatenated, with a concatenation dimension of 2.

[0049] 14) F3 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 2048, 2048 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses ReLU activation, and has an output size of 2048 and 2048 output channels. The average pooling layer has an input size of 2048, a kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 2048.

[0050] 15) Add the output of FC2 to the original feature.

[0051] 16) FC5 fully connected layer: input size 2048, output size 512, with additional weighting.

[0052] 17) FC6 fully connected layer: input size 512, output size 128, with additional weighting.

[0053] 18) Output layer: Input size 128, output size 1, with additional weighting.

[0054] 19) The result of C1 and the output of FC2 are concatenated, with a concatenation dimension of 2.

[0055] 20) F4 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses the ReLU activation function, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024.

[0056] 21) FC5 fully connected layer: input size 1024, output size 256, with additional weighting.

[0057] 22) FC6 fully connected layer: input size 256, output size 64, with additional weighting.

[0058] 23) Adaptive k-output layer: input size 64, output size 1, with additional weighting.

[0059] 3. Video anomaly detection algorithm based on MSGRU-TRF

[0060] 1) Training process of MSGRU-TRF multi-event learning algorithm

[0061] like Figure 3 As shown, the specific training process of the weakly supervised video anomaly detection algorithm based on multi-event learning implemented in this invention using the MSGRU-TRF network is as follows:

[0062] ① Select the preprocessed video feature vector and temporal resolution information as input, and perform video-level weakly supervised anomaly annotation on the video. The specific formula for calculating temporal resolution is as follows:

[0063]

[0064] In formula (1), X u Let N be the number of frames in the video, N be the number of segments in the video, and P be the number of frames per segment after standardizing the feature length. In this invention, N = 32 and P = 16.

[0065] ② The video feature vector is input into the GRU-based multi-scale temporal feature learning module of the MSGRU-TRF network, and the temporal resolution information is also input into the GRU-based multi-scale temporal feature learning module of the MSGRU-TRF network. The video feature vector is fused with the temporal resolution information through the output of FC2 after passing through the multi-scale GRU and two feature fusions F1 and F2. After residual and dimensionality reduction operations, the segment-level anomaly score vector used to calculate the loss function is obtained.

[0066] ③ Compare the difference between the anomaly score vector and the weakly supervised annotation. This invention uses the Top-k loss function to compare the difference between the generated anomaly scores and the weakly supervised annotation, and adjusts the network parameters during training by maximizing the score difference between anomaly segments and normal segments. The specific calculation formula is as follows:

[0067]

[0068] In formula (2), S and Y are the anomaly score vector and the weakly supervised annotation vector, respectively. BCE(S,Y) represents the cross-entropy loss calculated on S and Y. λ1 is a hyperparameter, and k is the number of segments in the anomaly video most likely to contain anomalies, generated by the adaptive k module. The score represents the segment of the normal video.

[0069] The score represents the segment score of the abnormal video. In this invention, λ1 = 0.9.

[0070] ④ To further optimize the reduced training flexibility caused by using a fixed k value in the Top-k loss function, an adaptive k module is introduced. The video feature vector after passing through C1 and the temporal resolution information after passing through FC2 are fused together and input into the dimensionality reduction layers FC5 and FC6. After dimensionality averaging, the k value used in the calculation in formula (2) is obtained. This module customizes a more suitable k value for the video based on both video features and temporal resolution information, increasing training flexibility and improving the model's detection capability.

[0071] ⑤ To constrain the generated k value in ④ and prevent the generation of extreme k values, this invention introduces an adaptive k-loss. A constant 3 (kMatrix) matrix with the same shape as the output features of the adaptive k-module is used to calculate the mean squared error with the output features of the adaptive k-module. By minimizing this loss, the generated k is made closer to 3, preventing extreme k values ​​from affecting the training process and increasing the robustness of model training. The specific calculation formula is as follows:

[0072]

[0073] In formula (3), N is the number of segments in the video. This represents the output fragment vector of adaptive k. Represents a vector of a constant matrix segment.

[0074] ⑥ Add formulas (2) and (3) together and incorporate the smoothing loss component to form the final multi-instance learning loss function (MILrankingLoss). This loss aims to reduce the gap between the abnormal scores generated by the network and the weakly supervised labels through training, while increasing the difference in abnormal scores between normal and abnormal videos, thereby improving the model's ability to identify abnormal segments. The specific calculation formula is as follows:

[0075]

[0076] In formula (4), s i+1 The segment score representing the next segment, s i This represents the segment score of the current segment. λ2, λ3, and λ4 are preset hyperparameters; in this invention, λ2 is chosen to be 1 × 10⁻⁴. -3λ3=8×10 -4 λ4=8×10 -3 .

[0077] 2) Testing process based on the MSGRU-TRF anomaly detection algorithm

[0078] like Figure 4 As shown, the specific testing process of the weakly supervised video anomaly detection algorithm based on multi-event learning implemented by the MSGRU-TRF network in this invention is as follows:

[0079] ① The long videos in the test set are divided into single video frame images. After dividing the video into N segments, video features are generated through a pre-trained I3D backbone network, and the temporal resolution is calculated at the same time.

[0080] ② Input the video features and corresponding temporal resolution information into the trained MSGRU-TRF reconstruction network to obtain the module output (anomaly score vector). Expand the segment-level anomaly scores into frame-level anomaly scores and compare this value with the set threshold. If anomaly_scores ≥ threshold, the test frame is an anomalous video frame.

[0081] When anomaly_scores < threshold, the test frames are normal video frames. In this invention, threshold = 0.65 is used.

[0082] ③ Compare the results of the judgment with the frame-level Groundtruth vector to obtain the false positive rate and true positive rate of the video frame. The specific calculation formulas for the false positive rate and true positive rate are as follows:

[0083]

[0084]

[0085] In formula (5), TP is the number of abnormal frames judged as abnormal, and FP is the number of normal frames judged as abnormal.

[0086] In formula (6), TN is the number of normal frames judged as normal, and FP is the number of normal frames judged as abnormal.

[0087] ④ Based on the false positive rate and true positive rate obtained in ③, plot the receiver operating characteristic curve (ROC). By calculating the area under the ROC curve (AUC), an overall quantitative evaluation index of the accuracy of anomaly detection can be obtained.

[0088] ⑤ Repeat steps ②③④ for all test videos that have completed the preprocessing process to detect all abnormal frames in the test videos, thereby achieving video anomaly detection.

Claims

1. A video anomaly detection method based on MSGRU-TRF, characterized in that, Includes the following steps: 1) Preprocessing First, the original long video data is segmented into single video frame images; second, the segmented video frames are divided into 32 fixed segments; weak supervision annotation is performed on each video at the video level; finally, the motion features of the video are initially extracted using a pre-trained I3D feature extractor to obtain the feature vector required for network input; in addition, the temporal resolution information of the video is calculated as an additional input to the network. 2) Constructing an MSGRU-TRF network The constructed network structure is as follows: A multi-scale GRU temporal feature learning network consists of three dilated convolutional layers (DC1, DC2, DC3), one convolutional layer (C1), three GRU layers (GRU1, GRU2, GRU3), a first feature fusion F1, a first GRU layer (GRU4), a second feature fusion F2, a first GRU layer (GRU5), two fully connected layers (FC1, FC2), a third feature fusion F3, and the module output. A temporal resolution feature mapping module consists of two fully connected layers (FC5, FC6). An adaptive k-module consists of two fully connected layers (FC1, FC2), a fourth feature aggregation F4, and two fully connected layers (FC3, FC4), and the adaptive k-module output. Finally, a network MSGRU-TRF is formed, which learns by combining multi-scale temporal information and temporal resolution information based on GRU. 3) Implementing unsupervised video anomaly detection using MSGRU-TRF networks. Training process: The original long videos in the training set are segmented into single video frames. The segmented video frames are then divided into 32 fixed segments, and weak supervision annotations are performed at the segment level for each video. The motion features of the video are initially extracted using a pre-trained I3D feature extractor to obtain the feature vector required for network input. In addition, the temporal resolution information of the video is calculated as an additional input to the network. The preprocessed video features and temporal resolution features are input into the multi-scale GRU temporal feature learning module and temporal resolution feature mapping module of the MSGRU-TRF network, respectively. After passing through the multi-scale GRU temporal feature learning module, the video features will obtain a score matrix containing the anomaly scores of each segment. After passing through the temporal resolution feature mapping module and the adaptive k module, the temporal resolution information will obtain the k value used to calculate the Top-k loss function and the feature vector used to calculate the Adaptive-k loss function. The network is optimized by minimizing the difference between the mean of the k segments with the highest anomaly scores in the input anomalous video and the maximum anomaly score in the normal video. The adaptive k module uses a k value customized according to the specific temporal resolution information of the video to assist in the optimization of the loss function. Finally, the MSE loss function is calculated using the output features of the adaptive k module and the preset value features to limit the generation of extreme k values. Testing process: The original long videos in the test set are segmented into single video frames. A pre-trained I3D feature extractor is used to initially extract motion features from the videos, obtaining the feature vectors required for network input. These vectors are then directly input into a trained MSGRU-TRF network to obtain segment-level anomaly scores. The anomaly score of each frame within a segment is the same as the anomaly score of that segment. The anomaly scores are then compared with the ground truth... Truth anomaly labeling can calculate the overall anomaly detection accuracy (AUC) of the video; by comparing the difference between the anomaly score and the set threshold, it can be determined whether the input video frame is an anomaly. The generated anomaly scores are simultaneously calculated using cross-entropy loss with weakly supervised annotations and Top-k loss between videos, supplemented by a smoothing loss term. The anomaly detection threshold is set to 0.

65.

2. The method according to claim 1, characterized in that: 1) DC1, DC2, DC3 three-dilated one-dimensional convolutional layer: input size is 2048, input channel number is 2048, kernel size is 3, stride is 1, dilation rate is 4, 2, 1 respectively, edge padding size is 4, 2, 1 respectively, activation function is ReLU, output size is 512, output channel number is 512; 2) C1 one-dimensional convolutional layer: input size is 2048, input channels are 2048, kernel size is 1, stride is 1, no edge padding, activation function is ReLU, output size is 512, output channels are 512; 3) Three GRU layers: GRU1, GRU2, and GRU3: input size is 512, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, and dropout rate is 0; 4) The results obtained through GRU1 and GRU2 are concatenated, with a concatenation dimension of 2; 5) F1 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses ReLU activation, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a pooling kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024. 6) GRU4GRU layer: input size is 1024, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, dropout rate is 0; 7) Concatenate the results obtained through GRU4 with the results obtained through GRU3, with a concatenation dimension of 2; 8) F2 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses ReLU activation, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024. 9) GRU5GRU layer: input size is 1024, hidden layer size is 512, loop direction is unidirectional, hidden layer depth is 1, dropout rate is 0; 10) The temporal resolution information calculated in the preprocessing stage is copied in terms of feature dimensions. The copying dimension is 1, and the size after copying is 32. 11) FC1 fully connected layer: input size 1, output size 128, with additional weighting; 12) FC2 fully connected layer: input size 128, output size 512, with additional weighting; 13) The results from GRU1, GRU4, and GRU5 are concatenated with the output of FC2, with a concatenation dimension of 2; 14) F3 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 2048, 2048 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses ReLU activation, and has an output size of 2048 and 2048 output channels. The average pooling layer has an input size of 2048, a pooling kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 2048. 15) Add the output of FC2 to the original feature; 16) FC5 fully connected layer: input size 2048, output size 512, with additional weighting; 17) FC6 fully connected layer: input size 512, output size 128, with additional weighting; 18) Output layer: Input size 128, output size 1, with additional weighting; 19) The result from C1 and the output from FC2 are concatenated, with a concatenation dimension of 2; 20) F4 Feature Fusion: Composed of a combination of one-dimensional convolutional layers and average pooling layers. The convolutional layer has an input size of 1024, 1024 input channels, a kernel size of 3, a stride of 1, an edge padding size of 1, uses ReLU activation, and has an output size of 1024 and 1024 output channels. The average pooling layer has an input size of 1024, a pooling kernel size of 3, a stride of 1, an edge padding size of 1, and an output size of 1024. 21) FC5 fully connected layer: input size 1024, output size 256, with additional weighting; 22) FC6 fully connected layer: input size 256, output size 64, with additional weighting; Adaptive k-output layer: input size 64, output size 1.

Images