Takifugu rubripes abnormal behavior identification method based on double correlation feature fusion

Abstract

The invention discloses a fugu rubripes abnormal behavior identification method based on double correlation feature fusion, and the method comprises the steps: obtaining video data of fugu rubripes, and determining the coordinate data of fish individuals in each video frame; drawing a prediction frame for the fish individuals in the high-level feature map set through the coordinate data to obtain a target high-level feature map set; and classifying and outputting the target high-level feature map set through Classification formed by a full connection layer. According to the invention, the accuracy of fish abnormal behavior identification is improved. Meanwhile, in a complex underwater environment, high similarity exists between abnormal behaviors of fugu rubripes and normal characteristics, and the success rate of abnormal recognition based on behavior pattern analysis is also improved.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and in particular to a method for identifying abnormal behavior of the redfin pufferfish based on dual correlation feature fusion. Background Technology

[0002] In aquaculture, the redfin pufferfish is an important economic fish species, and its health directly affects farming efficiency and product quality. Fish behavior analysis, as an effective means of assessing its health status, is of great significance for accurately detecting diseased redfin pufferfish underwater and for disease prevention. In recent years, photographic analysis of fish behavior has been commonly used; however, static images cannot comprehensively capture the movement trajectories and behavioral patterns of fish, thus reducing the accuracy of abnormal behavior identification. Furthermore, in the complex underwater environment, there is a high degree of similarity between abnormal behaviors and normal characteristics in redfin pufferfish, leading to a relatively high false positive rate in anomaly identification based on behavioral pattern analysis. Summary of the Invention

[0003] Based on this, it is necessary to propose an abnormal behavior identification method for redfin pufferfish based on dual correlation feature fusion to address the above problems.

[0004] A method for identifying abnormal behavior of the redfin pufferfish based on dual correlation feature fusion, the method comprising: Video data of the redfin pufferfish is acquired, and a first set of video frames is extracted from the video data using a first extraction method, and a second set of video frames is extracted from the video data using a second extraction method. The first set of video frames is denoised to obtain a set of denoised video frames. The coordinate data of individual fish in each video frame is determined based on the set of denoised video frames. The second video frame set is sparsely sampled to obtain a sparsely sampled video frame set, and the sparsely sampled video frame set is convolutionally pooled to obtain a first slow path video frame set. The second video frame set is subjected to dense sampling to obtain a densely sampled video frame set, and the densely sampled video frame set is subjected to convolutional pooling to obtain a first fast path video frame set. A first set of spatiotemporal feature maps, a second set of spatiotemporal feature maps, and a third set of spatiotemporal feature maps are determined based on the first set of slow path video frames and the first set of fast path video frames. The target slow path video frame set is determined based on the first slow path video frame set, and the target fast path video frame set is determined based on the first fast path video frame set. The first set of spatiotemporal feature maps, the second set of spatiotemporal feature maps, the third set of spatiotemporal feature maps, the target slow path video frame set, and the target fast path video frame set are added element by element to obtain the high-level feature map set; The target high-level feature map set is obtained by drawing prediction boxes for individual fish in the high-level feature map set using the coordinate data. The Classification layer, consisting of fully connected layers, classifies and outputs the set of high-level feature maps of the target.

[0005] In one embodiment, determining the coordinate data of an individual fish in each video frame based on the set of denoised video frames includes: The denoised video frame set is processed by the cross-stage feature coupling information bottleneck module to obtain a multi-scale feature map set. The multi-scale feature map set is sequentially subjected to global average pooling, one-dimensional convolution, and sigmoid function operations to obtain the channel feature map weight set. The weighted channel feature map set is obtained by multiplying the channel feature map weight set element by element with the multi-scale feature map set; Determine the first encoded output in the width direction and the second encoded output in the height direction of the pooling kernel for each weighted channel feature map in the set of weighted channel feature maps; Multiple first-encoded outputs are concatenated to obtain a height-encoded output, and multiple second-encoded outputs are concatenated to obtain a width-encoded output; The height encoding output and the width encoding output are concatenated, and then two-dimensional convolution and non-linear activation function operations are performed sequentially to obtain an intermediate feature map set; The weighted channel spatial feature map set is determined based on the intermediate feature map set; The coordinate data is obtained by processing the weighted channel spatial feature map set through the neck part of the YOLO model and the detection head.

[0006] In one embodiment, determining the weighted channel spatial feature map set based on the intermediate feature map set includes: The intermediate feature map set is split equally according to the number of channels to obtain a first intermediate feature map set and a second intermediate feature map set. The first intermediate feature map set and the second intermediate feature map set are then operated on by convolution transformation to obtain a height attention weight set and a width attention weight set, respectively. The spatial feature map weight set is determined based on the height attention weight in the height attention weight set and the width attention weight in the width attention weight set; The weighted channel spatial feature map set is obtained by multiplying the spatial feature map weight set and the weighted channel feature map set element by element.

[0007] In one embodiment, the first spatiotemporal feature map set is obtained through the following steps: Determine the dual-path high-order feature map set and the reconstructed feature map set based on the first slow path video frame set and the first fast path video frame set; After broadcasting the dual-path high-order feature map set along the channel dimension, it is added element by element to the reconstructed feature map set to obtain the first spatiotemporal feature map set; The second set of spatiotemporal feature maps is obtained through the following steps: The first slow path video frame set is convolved to obtain the second slow path video frame set. The first fast path video frame set is convolved to obtain the second fast path video frame set. A second spatiotemporal feature map set is determined based on the second slow path video frame set and the second fast path video frame set; The third set of spatiotemporal feature maps is obtained through the following steps: The second slow path video frame set is convolved to obtain the third slow path video frame set. The second fast path video frame set is convolved to obtain the third fast path video frame set. The third spatiotemporal feature map set is determined based on the third slow path video frame set and the third fast path video frame set.

[0008] In one embodiment, determining the target slow path video frame set based on the first slow path video frame set includes: The first slow path video frame set is convolved to obtain the second slow path video frame set, the second fast path video frame set is convolved to obtain the third fast path video frame set, the third fast path video frame set is convolved to obtain the fourth fast path video frame set, and the fourth fast path video frame set is convolved to obtain the fifth fast path video frame set. Determining the target fast path video frame set based on the first fast path video frame set includes: The first fast path video frame set is convolved to obtain the second fast path video frame set, the second fast path video frame set is convolved to obtain the third fast path video frame set, the third fast path video frame set is convolved to obtain the fourth fast path video frame set, and the fourth fast path video frame set is convolved to obtain the fifth fast path video frame set.

[0009] In one embodiment, the weighted channel feature map set is obtained by the following expression: Among them, M c denoted as the set of channel feature map weights; F is the set of multi-scale feature maps; G represents global average pooling; C1 represents one-dimensional convolution; σ represents the sigmoid function. This represents the set of weighted channel feature maps.

[0010] In one embodiment, the intermediate feature map set is obtained by the following expression: in, ϵF' is the feature map of the c-th weighted channel in The first feature point at the location; ϵF' is the feature map of the c-th weighted channel in The second feature point at the location; (h) represents the first encoded output; hϵ[0,H]; (w) represents the second encoded output; wϵ[0,W]; H represents the pooling kernel height; W represents the pooling kernel width; f represents the intermediate feature map set; Indicates the height-encoded output; represents the width-encoded output; [·,·] represents the concatenation operation; C2 represents two-dimensional convolution; δ is a non-linear activation function.

[0011] In one embodiment, the weighted channel spatial feature map set is obtained by the following expression: Among them, M s This is a set of weights for spatial feature maps; Weights based on high attention levels; is the width attention weight; (h,w) are the coordinates; M(F) is the set of weighted channel spatial feature maps; This is a set of weighted channel feature maps.

[0012] In one embodiment, the set of high-order feature maps for the dual-path approach is obtained by the following expression: Where Activity represents the activity parameter; Mobility represents the mobility parameter; SP represents the adaptive average pooling operation; σ² represents the variance calculation operation; F slow1 Represents the set of video frames in the first slow path; Concat represents the concatenation operation along the channel dimension; Conv and ReLU form a convolution-activation block; H slow H is a set of high-order feature maps for slow paths. fast H is the set of high-order feature maps for the fast path; H is the set of high-order feature maps for the dual path.

[0013] In one embodiment, the reconstructed feature map set is obtained by the following expression: Among them, F A For the fusion of feature maps; F slow1 For the first slow path video frame set; F fast1 For the first fast path video frame set; F T STPool is the set of feature weight thresholds; σ is the average pooling along the spatiotemporal dimension; Threshold represents the threshold function; ω SS This represents the set of strong features for slow paths; ω SW ω represents the set of weak features for slow paths. FS This is the set of strong features for fast paths; ω FW The set of weak features for fast paths; ω slow For the set of slow path weights; ω fast F is the set of fast path weights; W To reconstruct the set of weak features; F S is for reconstructing a strong feature set; 3D-SENet is a 3D SENet attention mechanism operation; P is the set of reconstructed feature maps.

[0014] This application improves the accuracy of identifying abnormal fish behavior. Furthermore, in complex underwater environments, the abnormal behavior of the red-finned pufferfish exhibits a high degree of similarity to normal characteristics, which also enhances the success rate of anomaly identification based on behavioral pattern analysis. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] in: Figure 1 This is an application environment diagram of a method for identifying abnormal behavior of redfin pufferfish based on dual correlation feature fusion in one embodiment. Figure 2 This is a flowchart of a method for identifying abnormal behavior of redfin pufferfish based on dual correlation feature fusion in one embodiment; Figure 3 This is a structural diagram of a method for identifying abnormal behavior of redfin pufferfish based on dual correlation feature fusion in one embodiment; Figure 4 This is a structural block diagram of obtaining a set of weighted channel spatial feature maps in one embodiment; Figure 5 Here is a structural block diagram of obtaining the reconstructed feature map in one embodiment; Figure 6 This is a comparison chart of the detection performance of this application and the baseline model in one embodiment; Figure 7 This is a schematic diagram of the normalized confusion matrix of behavior recognition results in one embodiment; Figure 8 This is a schematic diagram of the optical flow preprocessing process in one embodiment; Figure 9 This is a comparison chart of the recognition performance of different numbers of DCFFs in one embodiment; Figure 10 A visualization of partial features of the slow and fast paths in the fourth convolutional stage in one embodiment; Figure 11 This is a structural block diagram of a DCFF in one embodiment; Figure 12 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In aquaculture, the redfin pufferfish is an important economic fish species, and its health directly affects aquaculture efficiency and product quality. Fish behavior analysis, as an effective means of assessing its health status, is of great significance for accurately detecting diseased redfin pufferfish underwater and for disease prevention. In recent years, images have been commonly used to analyze fish behavior; however, static images cannot comprehensively capture the movement trajectory and behavioral patterns of fish, thus reducing the accuracy of abnormal behavior identification. Furthermore, in the complex underwater environment, there is a high similarity between abnormal behaviors and normal characteristics in redfin pufferfish, leading to a high false positive rate in anomaly identification based on behavioral pattern analysis. To address these technical problems, this application provides a method for identifying abnormal behaviors in redfin pufferfish based on dual correlation feature fusion.

[0019] Figure 1 This is an application environment diagram of a method for identifying abnormal behavior in the red-finned pufferfish based on dual correlation feature fusion, as shown in one embodiment. (Refer to...) Figure 1This method for identifying abnormal behavior in the red-finned pufferfish based on dual correlation feature fusion is applied to a system for identifying abnormal behavior in the red-finned pufferfish based on dual correlation feature fusion. This system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. Terminal 110 is used to acquire video data of redfin pufferfish, and extract a first set of video frames from the video data using a first extraction method, and extract a second set of video frames from the video data using a second extraction method; server 120 is used to perform denoising processing on the first set of video frames to obtain a denoised video frame set, and determine the coordinate data of individual fish in each video frame based on the denoised video frame set; sparsely sample the second set of video frames to obtain a sparsely sampled video frame set, and perform convolutional pooling on the sparsely sampled video frame set to obtain a first slow path video frame set; densely sample the second set of video frames to obtain a densely sampled video frame set, and perform convolutional pooling on the densely sampled video frame set to obtain a first fast path video frame set; according to the first... The slow path video frame set and the first fast path video frame set determine the first spatiotemporal feature map set, the second spatiotemporal feature map set, and the third spatiotemporal feature map set; the target slow path video frame set is determined based on the first slow path video frame set, and the target fast path video frame set is determined based on the first fast path video frame set; the first spatiotemporal feature map set, the second spatiotemporal feature map set, the third spatiotemporal feature map set, the target slow path video frame set, and the target fast path video frame set are added element-wise to obtain the high-level feature map set; prediction boxes are drawn for individual fish in the high-level feature map set using the coordinate data to obtain the target high-level feature map set; the target high-level feature map set is classified and output using Classification composed of fully connected layers.

[0020] like Figure 2 and Figure 3 As shown, in one embodiment, a method for identifying abnormal behavior of the red-finned pufferfish based on dual correlation feature fusion is provided. This method can be applied to both terminals and servers; this embodiment illustrates its application to a terminal. The specific steps of this method for identifying abnormal behavior of the red-finned pufferfish based on dual correlation feature fusion are as follows: S10: Acquire video data of the redfin pufferfish, and extract a first set of video frames from the video data using a first extraction method, and extract a second set of video frames from the video data using a second extraction method; S20: Denoise the first video frame set to obtain a denoised video frame set, and determine the coordinate data of the individual fish in each video frame based on the denoised video frame set; S30: Sparsely sample the second video frame set to obtain a sparsely sampled video frame set, and then perform convolutional pooling on the sparsely sampled video frame set to obtain a first slow path video frame set. S40: Perform dense sampling on the second video frame set to obtain a densely sampled video frame set, and perform convolutional pooling on the densely sampled video frame set to obtain a first fast path video frame set. S50: Determine a first spatiotemporal feature map set, a second spatiotemporal feature map set, and a third spatiotemporal feature map set based on the first slow path video frame set and the first fast path video frame set; S60: Determine the target slow path video frame set based on the first slow path video frame set, and determine the target fast path video frame set based on the first fast path video frame set. S70: The first set of spatiotemporal feature maps, the second set of spatiotemporal feature maps, the third set of spatiotemporal feature maps, the target slow path video frame set, and the target fast path video frame set are added element by element to obtain a high-level feature map set; S80: Using the coordinate data, draw prediction boxes for individual fish in the high-level feature map set to obtain the target high-level feature map set; S90: The target high-level feature map set is classified and output through Classification, which consists of fully connected layers, and the classification result is normal or abnormal.

[0021] In one embodiment, determining the coordinate data of individual fish in each video frame based on the denoised video frame set in step S20 includes: S201: The denoised video frame set is processed by the cross-stage feature coupling information bottleneck module to obtain a multi-scale feature map set; S202: Perform global average pooling, one-dimensional convolution, and sigmoid function operations sequentially on the multi-scale feature map set to obtain the channel feature map weight set; S203: Multiply the channel feature map weight set with the multi-scale feature map set element by element to obtain a weighted channel feature map set; S204: Determine the first encoded output in the width direction and the second encoded output in the height direction of the pooling kernel for each weighted channel feature map in the set of weighted channel feature maps; S205: Concatenate multiple first encoding outputs to obtain a height encoding output, and concatenate multiple second encoding outputs to obtain a width encoding output; S206: After concatenating the height encoding output and the width encoding output, perform two-dimensional convolution operation and non-linear activation function operation in sequence to obtain an intermediate feature map set; S207: Determine the weighted channel spatial feature map set based on the intermediate feature map set; S208: The coordinate data is obtained by processing the weighted channel spatial feature map set through the neck part of the YOLO model and the detection head.

[0022] In one embodiment, determining the weighted channel spatial feature map set based on the intermediate feature map set in step S207 includes: S2071: The intermediate feature map set is split equally according to the number of channels to obtain a first intermediate feature map set and a second intermediate feature map set respectively. The first intermediate feature map set and the second intermediate feature map set are operated on by convolution transformation to obtain a height attention weight set and a width attention weight set respectively. S2072: Determine the spatial feature map weight set based on the height attention weights in the height attention weight set and the width attention weights in the width attention weight set; S2073: Multiply the spatial feature map weight set and the weighted channel feature map set element by element to obtain the weighted channel spatial feature map set.

[0023] In one embodiment, the first spatiotemporal feature map set in step S70 is obtained through the following steps: S701: Determine the dual-path high-order feature map set and the reconstructed feature map set based on the first slow path video frame set and the first fast path video frame set; S702: After broadcasting the dual-path high-order feature map set along the channel dimension, add it element by element to the reconstructed feature map set to obtain the first spatiotemporal feature map set; The second set of spatiotemporal feature maps in step S70 is obtained through the following steps: S703: Perform a convolution operation on the first slow path video frame set to obtain a second slow path video frame set. S704: Perform a convolution operation on the first fast path video frame set to obtain the second fast path video frame set; S705: Determine the second spatiotemporal feature map set based on the second slow path video frame set and the second fast path video frame set; The third spatiotemporal feature map set mentioned in step S70 is obtained through the following steps: S706: Perform a convolution operation on the second slow path video frame set to obtain a third slow path video frame set; S707: Perform a convolution operation on the second fast path video frame set to obtain the third fast path video frame set; S708: Determine the third spatiotemporal feature map set based on the third slow path video frame set and the third fast path video frame set.

[0024] In one embodiment, determining the target slow path video frame set based on the first slow path video frame set in step S60 includes: S601: Perform a convolution operation on the first slow path video frame set to obtain a second slow path video frame set, perform a convolution operation on the second fast path video frame set to obtain a third fast path video frame set, perform a convolution operation on the third fast path video frame set to obtain a fourth fast path video frame set, and perform a convolution operation on the fourth fast path video frame set to obtain a fifth fast path video frame set. Determining the target fast path video frame set based on the first fast path video frame set in step S60 includes: S602: Perform a convolution operation on the first fast path video frame set to obtain a second fast path video frame set, perform a convolution operation on the second fast path video frame set to obtain a third fast path video frame set, perform a convolution operation on the third fast path video frame set to obtain a fourth fast path video frame set, and perform a convolution operation on the fourth fast path video frame set to obtain a fifth fast path video frame set.

[0025] Specifically, in the target localization stage, each video segment needs to pass through a target detection network to obtain fish coordinate data. The accuracy and detection speed of this network directly impact the performance of subsequent fish behavior classification. The YOLO series, as an advanced algorithm in single-stage detection, completes target regression and classification simultaneously through a single network, greatly simplifying the detection process and improving processing speed. Considering that YOLOv12 is relatively new and may have compatibility issues on some older hardware or software platforms, this study designs the ECAM attention mechanism based on YOLOv10 and proposes an enhanced target detection network, EYOLO, to focus on key target features, reduce background interference, and enhance the localization capability for complex underwater fish individuals.

[0026] In the behavior recognition stage, the behavior of the red-finned pufferfish is not obvious, resulting in high similarity between different behaviors, which poses a challenge to the identification of abnormal behaviors. Although the Transformer-based model has a powerful global receptive field, it is not sensitive enough to behavioral details and relies on a large amount of training data, thus limiting its recognition performance. Inspired by the human visual system, the SlowFast model is based on a two-stream network design, using fast and slow paths to process the temporal and spatial dimensions of the video, respectively. The high frame rate processing of the fast path enables the model to capture subtle changes in fish behavior. The network structure parameters are shown in Table 1 (REF _Ref1596), with the convolution kernel dimension being {T×C×S²}, where C, T, and S represent the channel, temporal, and spatial sizes, respectively. The stride is represented as {T×S²}, where T and S are the temporal and spatial strides, respectively. Therefore, this study, based on the SlowFast model, optimizes the fusion of spatiotemporal features by designing a DCFF mechanism, thereby achieving accurate identification of abnormal behaviors in the red-finned pufferfish.

[0027] In one embodiment, to more accurately locate individual fish, global pooling or a fixed-size sliding window is typically used when processing the target's spatial location information. However, these operations may overlook important details in the image. Therefore, spatial attention is decomposed into two one-dimensional feature encoding processes, aggregating features along two spatial directions respectively, to capture long-range dependencies while maintaining accurate location information. Figure 4 As shown, the weighted channel feature map set is obtained through the following expression: (1) (2) Among them, M c Represents the set of channel feature map weights; F∈R H×W×C It is a set of multi-scale feature maps; G represents global average pooling; C1 represents one-dimensional convolution; σ represents the sigmoid function; This represents the set of weighted channel feature maps.

[0028] In one embodiment, pooling kernels of sizes (H,1) and (1,W) are first used to encode the feature map of each channel along the horizontal and vertical directions, respectively; the intermediate feature map set is obtained by the following expression: (3) (4) (5) in, ϵF' is the feature map of the c-th weighted channel in The first feature point at the location; ϵF' is the feature map of the c-th weighted channel in The second feature point at the location; (h) represents the first encoded output; hϵ[0,H]; (w) represents the second encoded output; wϵ[0,W]; H represents the pooling kernel height; W represents the pooling kernel width; f represents the intermediate feature map set; Indicates the height-encoded output; represents the width-encoded output; [·,·] represents the concatenation operation; C2 represents two-dimensional convolution; δ is a non-linear activation function.

[0029] In one embodiment, the weighted channel spatial feature map set is obtained by the following expression: (6) (7) Among them, M s This is a set of weights for spatial feature maps; Weights based on high attention levels; is the width attention weight; (h,w) are the coordinates; M(F) is the set of weighted channel spatial feature maps; This is a set of weighted channel feature maps.

[0030] In one embodiment, to account for the changes in anomalous behavior over time and thus amplify the differences between different behaviors, traditional methods typically perform average pooling of feature maps from each stage of the backbone network along the spatiotemporal dimension, and then concatenate them along the channel dimension as feature input for the final classification task. However, this method only considers the average information content of the features in the time dimension, i.e., it is limited to the first-order representation of the features. To perform higher-order temporal modeling of the features, activity parameters and mobility parameters are introduced to capture the changes in fish behavior in the video. Activity parameters reflect the stability or trend of fish movement over a period of time, while mobility parameters reveal more about the intensity of the fish's movement at a particular moment. Figure 5 As shown, the set of high-order feature maps for the dual-path system is obtained through the following expression: (8) (9) To further enhance the temporal information representation capability of different feature points, activity parameters and mobility parameters are concatenated to generate a set of high-order feature maps H for slow paths. slow The process is shown in equation (10). Simultaneously, the fast path feature map is processed in the same way to obtain the corresponding set of high-order fast path feature maps H. fastand with the set of high-order feature maps H of the slow path slow The two paths are concatenated to obtain a set of high-order feature maps H. The concatenation process is shown in equation (11). (10) (11) Where Activity represents the activity parameter; Mobility represents the mobility parameter; SP represents the adaptive average pooling operation; σ² represents the variance calculation operation; F slow1 Represents the set of video frames in the first slow path; Concat represents the concatenation operation along the channel dimension; Conv and ReLU form a convolution-activation block; H slow H is a set of high-order feature maps for slow paths. fast H is the set of high-order feature maps for the fast path; H is the set of high-order feature maps for the dual path.

[0031] In one embodiment, for the obtained first slow path video frame set F slow1 ∈R T×C×H×W and the first fast path video frame set F fast1 ∈R αT×βC×H×W To match the size of the feature maps, the fast path is first subjected to feature morphology transformation, compressing all α frames into a single frame channel, i.e., {αT×βC×H×W} is reconstructed into {T×αβC×H×W}. Then, to separate features according to richness, the feature maps of the two paths are first added element-wise to obtain the fused feature map set F. A Then, after pooling along the spatiotemporal dimension, the feature weight threshold set F is obtained. T The process is shown in equations (12)-(13).

[0032] The reconstructed feature map set is obtained through the following expression: (12) (13) The fast and slow paths are processed separately using batch normalization (BN) and the sigmoid function to generate a unique set of slow path weights ω at each spatiotemporal location. slow and the set of fast path weights ω fast By comparing the weight information at different stages with the feature weight threshold, accurate separation of strong and weak features at different layers can be achieved.

[0033] (14) (15) To further promote feature interaction between different branches, the strong and weak information of each level in the two paths are combined respectively. The specific process is shown in equations (16)-(18), and the weak feature set F is reconstructed. W and reconstruct the strong feature set F S Let F represent the weak and strong features generated after reconstruction, respectively. This is to enhance the reconstructed weak feature set F. W To improve utilization, the 3D-SENet attention mechanism is used to reconstruct the weak feature set F. W The features are then reweighted. 3D-SENet extracts and reconstructs the weak feature set F through compression operations. W The overall information in the dataset is extracted, and channel weights are generated using expansion operations to achieve adaptive readjustment between weak feature channels. Finally, the result processed by 3D-SENet is compared with the reconstructed strong feature set F. S The features are combined to construct a more comprehensive feature representation. The process of separating and reconstructing strong and weak features highlights the key features of abnormal behavior, enhances the feature boundaries between different behaviors, and is conducive to the accurate identification of abnormal behavior of underwater red-finned pufferfish by deep networks.

[0034] (16) (17) (18) Among them, F A For the fusion of feature maps; F slow1 For the first slow path video frame set; F fast1 For the first fast path video frame set; F T STPool is the set of feature weight thresholds; σ is the average pooling along the spatiotemporal dimension; Threshold represents the threshold function; ω SS This represents the set of strong features for slow paths; ω SW ω represents the set of weak features for slow paths. FS This is the set of strong features for fast paths; ω FW The set of weak features for fast paths; ω slow For the set of slow path weights; ω fast F is the set of fast path weights; W To reconstruct the set of weak features; F S is for reconstructing a strong feature set; 3D-SENet is a 3D SENet attention mechanism operation; P is the set of reconstructed feature maps.

[0035] This application improves the accuracy of identifying abnormal fish behavior. Furthermore, in complex underwater environments, the abnormal behavior of the red-finned pufferfish exhibits a high degree of similarity to normal characteristics, which also enhances the success rate of anomaly identification based on behavioral pattern analysis.

[0036] 1. Model Training and Evaluation 1.1 Experimental Platform and Hyperparameter Settings To ensure the accuracy and reliability of the experimental data, the training and testing of the underwater fish behavior recognition model were conducted in the same environment. The experimental environment is shown in Table 2 of REF_Ref17770.

[0037] Experimental hyperparameter settings: batch size of 8, training cycle of 300 rounds, Adam optimizer, momentum factor of 0.9, initial learning rate of 0.001, and weight decay coefficient of 0.0002.

[0038] 1.2 Evaluation Indicators In the field of computer vision, evaluating the detection performance of a model is crucial for assessing its quality. Considering that a single metric is often insufficient to comprehensively evaluate a model's performance, precision (P), recall (R), and F1 score (F1) are introduced to assess the model's recognition ability from different perspectives. Precision refers to the proportion of samples predicted as positive by the model that are actually positive, reflecting the reliability of the model's positive predictions. Recall refers to the proportion of all samples that are actually positive that are correctly predicted as positive by the model, indicating the model's ability to identify all positive samples. The F1 score is the harmonic mean of precision and recall, used to comprehensively measure the model's performance. The calculation formulas are as follows (12)-(14): (19) (20) (twenty one) The expression `mAP@0.5` represents the average accuracy with an IoU threshold of 0.5. Specifically, `TP` represents the number of samples correctly predicted as fish, `FP` represents the number of samples incorrectly predicted as fish, `FN` represents the number of samples not predicted as fish, and `mAP@0.5` represents the average accuracy with an IoU threshold of 0.5.

[0039] 2. Experimental Results and Analysis 2.1 Ablation Experiment To verify the effectiveness of the improved model, a series of ablation experiments were designed using the SlowFast network as the baseline method. First, edge preprocessing, ECAM, and DCFF modules were added to the baseline model to investigate the impact of each module on the baseline model's performance. After introducing individual modules, the synergistic effects between these modules were further explored. The three modules were combined in pairs and added to the baseline model to investigate the impact of different module combinations on the baseline model. Finally, all modules were added to the baseline model simultaneously to construct the proposed model DCFF-EYSFNet. The experimental results are shown in Table 3 (REF _Ref15641 \h).

[0040] As a baseline model, SlowFast achieved 82.5% precision, 79.8% recall, and 81.1% F1 score in the tests. While its unique dual-stream path design allows for the separate extraction of temporal and spatial features from videos, and the fusion of temporal and spatial streams at different network stages, this fusion mechanism primarily focuses on using behavioral features to facilitate the extraction of appearance features. However, the appearance and behavioral features of the underwater red-finned pufferfish are often not simultaneously apparent, which limits its ability to utilize both spatiotemporal features, thus limiting its application in underwater fish behavior recognition tasks. In the experimental results, the addition of ECAM significantly improved the model's accuracy, reaching 86.8%. This improvement is mainly attributed to ECAM enabling the model to focus more on the area where the individual fish is located, enhancing the feature representation of the target object. This helps the model better distinguish the feature differences between the foreground and background, effectively filtering out background information interference and improving fish localization accuracy. However, EYOLO did not significantly improve recall. This is because juvenile redfin pufferfish are relatively small, and with increasing network depth, some marginal or indistinct features cannot be accurately extracted due to information loss, resulting in the failure to correctly detect abnormal behaviors in some individuals. Furthermore, the addition of the DCFF module improved the baseline model by 4.2% in precision and 5.6% in recall. This is thanks to the Hjorth module in DCFF, which captures changes in different feature points along the time dimension, especially for individuals with less obvious behavior. Meanwhile, PFR, after separating the spatiotemporal information of fish strength, reduces the extraction of irrelevant information, enhances the correlation between the two paths, and preserves rich features to the greatest extent.

[0041] By preprocessing the baseline model and simultaneously incorporating both ECAM and DCFF, optimal performance was achieved in identifying anomalous behavior, with accuracy, recall, and F1 score reaching 90.3%, 87.4%, and 88.8%, respectively. Figure 6As shown, the comparison of the recognition performance of the proposed model and the baseline model in the same video frame is presented. The first row shows the recognition results of the video frame after cropping the water tank background, and the second row is a magnified view of the lower left corner of the video frame. The baseline model exhibits uncertainty in its behavior classification results in the video frame, displaying both "normal" and "abnormal" labels simultaneously. In contrast, the proposed model detects only "abnormal" labels for the same individual in the same video frame, with a high confidence level. This further validates the effectiveness and superiority of the proposed model in recognizing abnormal behaviors of the red-finned pufferfish.

[0042] 2.2 Comparative Experiment To verify the superior performance of the proposed model, comparative experiments were conducted using mainstream video recognition models and models that currently demonstrate significant effectiveness in underwater fish behavior recognition. All models were trained and tested on the same dataset to ensure the fairness and comparability of the experimental results. The experimental results are shown in Table 4 of REF_Ref19926.

[0043] Compared to traditional 3D convolutional video recognition models C3D and I3D, DCFF-EYSFNet significantly improves the F1 score, by 11.9% and 6.3%, respectively. This is attributed to the limitations of C3D and I3D models in spatiotemporal feature extraction. Specifically, C3D uses fixed-size and fixed-shape 3D convolutional kernels to extract spatiotemporal features, which struggles to capture motion at different scales and speeds. Furthermore, its fixed network structure lacks an adaptive mechanism, failing to adjust processing strategies based on the quality and content of the input video. In underwater videos, variations in light and water quality can significantly impact video quality, exacerbating the performance instability of C3D. In contrast, I3D introduces optical flow into its two-stream structure, enabling the model to more accurately capture dynamic changes in the video, thus outperforming C3D. However, optical flow extraction in I3D is limited by the complex underwater environment; turbid water degrades optical flow quality, limiting the accuracy of fish behavior recognition.

[0044] Compared to current state-of-the-art spatiotemporal detection models Video Swin-T and YOWOv2, DCFF-EYSFNet achieves higher recall rates of 4.9% and 10.8% respectively, demonstrating its advantage in reducing missed detections of fish behaviors, resulting in fewer missed abnormal behaviors. While Video Swin-T and YOWOv2 show good performance in video behavior recognition, their performance is limited in the specific task of identifying abnormal behaviors in the underwater red-finned pufferfish. Specifically, YOWOv2, through its multi-level detection channels, can detect behavioral features of targets of different sizes, enhancing the identification of juvenile fish to some extent, achieving accuracy and recall rates of 85.8% and 82.5% respectively. However, the similarity of different behavioral features in juvenile fish masks differences in target size, making it difficult for YOWOv2 to effectively distinguish between them. The Video Swin-T model employs window partitioning and moving strategies to efficiently extract features, but when dealing with targets with indistinct details or similar behavioral patterns, its local feature extraction method may not be able to fully capture all the information about abnormal behavior. Furthermore, the model is relatively weak in modeling long-term dependencies, which limits its performance in complex behavior recognition tasks.

[0045] ST-GCN, as a superior underwater fish behavior recognition model, continuously tracks the position of fish in videos and transforms the tracking results into graph convolutional structured data, thereby quantifying individual abnormal behaviors and achieving an accuracy of 83.3% in behavior classification. However, this model relies on accurate annotation of fish joints to construct skeleton sequences, which limits its applicability to different species and sizes of fish. In contrast, the proposed DCFF-EYSFNet model demonstrates the most outstanding performance, achieving accuracy of 90.3%, recall of 87.4%, and F1 score of 88.8%. Figure 7 The figure shows the normalized confusion matrix of the behavior recognition results. This matrix demonstrates the accuracy of the proposed model in recognizing normal and abnormal behaviors. In particular, the model achieves an accuracy of 97% when recognizing normal fish. For abnormal fish, the proportion of correctly identified behaviors is 85%, while the proportion incorrectly identified as normal behaviors is only 1%, indicating that the proposed model has an extremely low false negative rate in recognizing fish anomalies. Therefore, DCFF-EYSFNet has significant advantages in underwater fish behavior recognition tasks.

[0046] 2.3 Experiments with different pretreatment methods In computer vision tasks, preprocessing steps are crucial for improving the accuracy and efficiency of models. To explore the differences in how different preprocessing strategies enhance model detection performance, two preprocessing methods were designed: edge preprocessing and optical flow preprocessing. Edge preprocessing employs the Canny edge detection algorithm to extract the contours of targets in video frames and fuse them with the original image to enhance the edge features of the targets. Optical flow preprocessing focuses on the motion information of pixels in the video sequence, extracting optical flow information by calculating the optical flow field between adjacent frames and combining it with the original video frames. To evaluate the specific impact of these two preprocessing strategies on model detection performance, comparative experiments were designed and implemented to analyze the advantages and disadvantages of each method and its applicability in different application scenarios. The experimental results are shown in Table 5 (REF _Ref22933 \h).

[0047] Compared to the baseline model, the SlowFast model with edge preprocessing achieved accuracy, recall, and F1 score of 83.2%, 80.0%, and 81.6%, respectively, representing improvements of 0.7%, 0.2%, and 0.5%. This indicates that the key structural features of fish contained in the edge information, when fused with the original image, can significantly enhance the pose representation of the fish. By highlighting the contours and morphological details of the fish, it provides the model with richer spatial structural cues, thereby improving the accuracy of subsequent target localization tasks. When optical flow preprocessing was integrated into the baseline model, drastically different results emerged: the model's accuracy, recall, and F1 score all decreased compared to the baseline model, especially the accuracy, which dropped by 5.7%. This is because the integration of optical flow maps inevitably alters the texture and color features of the original image, interfering with the model's fine recognition of individual fish features. Furthermore, the application of optical flow methods also has limitations when most individuals in a fish group are stationary, such as... Figure 8 As shown, an individual in a stationary state failed to effectively extract the corresponding optical flow.

[0048] Given the limitations of optical flow preprocessing techniques in static scenes and their potential interference with the features of the original image, this study extracts only the contour information of individual fish and fuses it with the original image. This method preserves key clues of fish movement while maintaining the integrity of its external features, which is beneficial for the accuracy of subsequent localization of individual fish.

[0049] 2.4 Number of DCFF modules like Figure 11To effectively promote deep interaction between spatiotemporal features of the two paths at different stages, a DCFF feature fusion mechanism was designed to flexibly adapt to feature information at different levels, thereby achieving more refined and efficient feature fusion. To investigate the impact of different numbers of DCFFs on the final detection accuracy of the model, an experiment was designed to select the optimal number of DCFFs.

[0050] Experimental results are as follows Figure 9 As shown in the figure, this graph visually illustrates the changing trends of various detection metrics of the model under different numbers of DCFF modules. With the increase in the number of DCFF modules, the model's precision, recall, and F1 score generally show a trend of first increasing and then decreasing. Specifically, when the number of DCFF modules increases from 1 to 3, the model's precision increases from 88.1% to 90.3%, and the recall increases from 85.5% to 87.4%. At a quantity of 3, all evaluation metrics reach their optimal levels. However, when the number of DCFF modules further increases to 4, although the model's evaluation metrics remain at a high level, there is a slight decline compared to the optimal state when the quantity is 3. Therefore, the number of DCFF modules is not necessarily better the more there are; rather, there is an optimal number.

[0051] The trends of various indicators show that the Hjorth module in DCFF can extract the correlation between spatiotemporal features in different stages, while the PFR module reduces irrelevant features in the two-path fusion process and maximizes the fusion of key spatiotemporal features of fish. This dual-correlation feature fusion mechanism realizes the interactivity of features at each stage of different paths. Therefore, appropriately increasing the number of DCFF modules can improve the model's feature fusion capability. To further explore the reason why increasing the number of DCFFs in the later stages of the network leads to performance degradation, the two-path feature extraction results of the Res4 stage of the network were visualized. Figure 10 As shown, both paths are already at the high-level feature stage in the Res4 network, each generating corresponding high-level semantic information based on shallow, low-level information. At this point, the network cannot effectively fuse the abstract information from different branches, and may even experience fusion conflicts due to differences in spatial and temporal features. Therefore, increasing the number of DCFFs within a certain range can indeed enhance the model's feature fusion capability. However, when the number of DCFFs exceeds a certain threshold, the model performance will tend to decline.

[0052] in conclusion This study proposes a novel fish behavior recognition method, DCFF-EYSFNet, based on dual-correlation feature fusion, aiming to address the misclassification problem in recognizing different behaviors of the red-finned pufferfish in underwater environments. In terms of implementation, the target localization and behavior classification processes are first decoupled to better train the model. Then, image processing techniques are used to extract and fuse the edge contours of individual fish in the raw video data to enhance the fish's pose information. To achieve continuous tracking of individual behaviors, an ECAM module is designed in the target localization network to focus on the key features of individual fish. In the behavior classification stage, a DCFF spatiotemporal feature fusion mechanism is proposed. This mechanism promotes the correlation between fast and slow paths in different stages, thereby amplifying the differences between different behaviors. Experimental results show that compared with the baseline model, the proposed model improves accuracy and recall by 7.8% and 7.6%, respectively, fully validating the effectiveness of the method. However, this decoupled method requires processing through two networks sequentially, increasing inference time and limiting the model's real-time performance and efficiency. Therefore, how to maintain high detection accuracy while making the network structure lightweight is a future research direction.

[0053] Figure 12 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 12 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a method for identifying abnormal behavior of the red-finned pufferfish based on dual-correlation feature fusion. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the same method. Those skilled in the art will understand that... Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0054] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0055] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0056] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for identifying abnormal behavior of the red-finned pufferfish based on dual correlation feature fusion, characterized in that, The method includes: Video data of the redfin pufferfish is acquired, and a first set of video frames is extracted from the video data using a first extraction method, and a second set of video frames is extracted from the video data using a second extraction method. The first set of video frames is denoised to obtain a set of denoised video frames. The coordinate data of individual fish in each video frame is determined based on the set of denoised video frames. The second video frame set is sparsely sampled to obtain a sparsely sampled video frame set, and the sparsely sampled video frame set is convolutionally pooled to obtain a first slow path video frame set. The second video frame set is subjected to dense sampling to obtain a densely sampled video frame set, and the densely sampled video frame set is subjected to convolutional pooling to obtain a first fast path video frame set. A first set of spatiotemporal feature maps, a second set of spatiotemporal feature maps, and a third set of spatiotemporal feature maps are determined based on the first set of slow path video frames and the first set of fast path video frames. The target slow path video frame set is determined based on the first slow path video frame set, and the target fast path video frame set is determined based on the first fast path video frame set. The first set of spatiotemporal feature maps, the second set of spatiotemporal feature maps, the third set of spatiotemporal feature maps, the target slow path video frame set, and the target fast path video frame set are added element by element to obtain the high-level feature map set; The target high-level feature map set is obtained by drawing prediction boxes for individual fish in the high-level feature map set using the coordinate data. The Classification layer, consisting of fully connected layers, classifies and outputs the set of high-level feature maps of the target.

2. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 1, characterized in that, The step of determining the coordinate data of individual fish in each video frame based on the set of denoised video frames includes: The denoised video frame set is processed by the cross-stage feature coupling information bottleneck module to obtain a multi-scale feature map set. The multi-scale feature map set is sequentially subjected to global average pooling, one-dimensional convolution, and sigmoid function operations to obtain the channel feature map weight set. The weighted channel feature map set is obtained by multiplying the channel feature map weight set element by element with the multi-scale feature map set; Determine the first encoded output in the width direction and the second encoded output in the height direction of the pooling kernel for each weighted channel feature map in the set of weighted channel feature maps; Multiple first-encoded outputs are concatenated to obtain a height-encoded output, and multiple second-encoded outputs are concatenated to obtain a width-encoded output; The height encoding output and the width encoding output are concatenated, and then two-dimensional convolution and non-linear activation function operations are performed sequentially to obtain an intermediate feature map set; The weighted channel spatial feature map set is determined based on the intermediate feature map set; The coordinate data is obtained by processing the weighted channel spatial feature map set through the neck part of the YOLO model and the detection head.

3. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 2, characterized in that, The intermediate feature map set determines the weighted channel spatial feature map set, including: The intermediate feature map set is split equally according to the number of channels to obtain a first intermediate feature map set and a second intermediate feature map set. The first intermediate feature map set and the second intermediate feature map set are then operated on by convolution transformation to obtain a height attention weight set and a width attention weight set, respectively. The spatial feature map weight set is determined based on the height attention weight in the height attention weight set and the width attention weight in the width attention weight set; The weighted channel spatial feature map set is obtained by multiplying the spatial feature map weight set and the weighted channel feature map set element by element.

4. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 1, characterized in that, The first set of spatiotemporal feature maps is obtained through the following steps: Determine the dual-path high-order feature map set and the reconstructed feature map set based on the first slow path video frame set and the first fast path video frame set; After broadcasting the dual-path high-order feature map set along the channel dimension, it is added element by element to the reconstructed feature map set to obtain the first spatiotemporal feature map set; The second set of spatiotemporal feature maps is obtained through the following steps: The first slow path video frame set is convolved to obtain the second slow path video frame set. The first fast path video frame set is convolved to obtain the second fast path video frame set. A second spatiotemporal feature map set is determined based on the second slow path video frame set and the second fast path video frame set; The third set of spatiotemporal feature maps is obtained through the following steps: The second slow path video frame set is convolved to obtain the third slow path video frame set. The second fast path video frame set is convolved to obtain the third fast path video frame set. The third spatiotemporal feature map set is determined based on the third slow path video frame set and the third fast path video frame set.

5. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 4, characterized in that, Determining the target slow path video frame set based on the first slow path video frame set includes: The first slow path video frame set is convolved to obtain the second slow path video frame set, the second fast path video frame set is convolved to obtain the third fast path video frame set, the third fast path video frame set is convolved to obtain the fourth fast path video frame set, and the fourth fast path video frame set is convolved to obtain the fifth fast path video frame set. Determining the target fast path video frame set based on the first fast path video frame set includes: The first fast path video frame set is convolved to obtain the second fast path video frame set, the second fast path video frame set is convolved to obtain the third fast path video frame set, the third fast path video frame set is convolved to obtain the fourth fast path video frame set, and the fourth fast path video frame set is convolved to obtain the fifth fast path video frame set.

6. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 3, characterized in that, The weighted channel feature map set is obtained through the following expression: Among them, M c denoted as the set of channel feature map weights; F is the set of multi-scale feature maps; G represents global average pooling; C1 represents one-dimensional convolution; σ represents the sigmoid function. This represents the set of weighted channel feature maps.

7. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 6, characterized in that, The intermediate feature map set is obtained through the following expression: in, ϵF' is the feature map of the c-th weighted channel in The first feature point at the location; ϵF' is the feature map of the c-th weighted channel in The second feature point at the location; (h) represents the first encoded output; hϵ[0,H]; (w) represents the second encoded output; wϵ[0,W]; H represents the pooling kernel height; W represents the pooling kernel width; f represents the intermediate feature map set; Indicates the height-encoded output; represents the width-encoded output; [·,·] represents the concatenation operation; C2 represents two-dimensional convolution; δ is a non-linear activation function.

8. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 7, characterized in that, The weighted channel spatial feature map set is obtained through the following expression: Among them, M s This is a set of weights for spatial feature maps; Weights based on high attention levels; is the width attention weight; (h,w) are the coordinates; M(F) is the set of weighted channel spatial feature maps; This is a set of weighted channel feature maps.

9. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 4, characterized in that, The set of high-order feature maps for dual paths is obtained through the following expression: Where Activity represents the activity parameter; Mobility represents the mobility parameter; SP represents the adaptive average pooling operation; σ² represents the variance calculation operation; F slow1 Represents the set of video frames in the first slow path; Concat represents the concatenation operation along the channel dimension; Conv and ReLU form a convolution-activation block; H slow H is a set of high-order feature maps for slow paths. fast H is the set of high-order feature maps for the fast path; H is the set of high-order feature maps for the dual path.

10. The method for identifying abnormal behavior of red-finned pufferfish based on dual correlation feature fusion according to claim 1, characterized in that, The reconstructed feature map set is obtained through the following expression: Among them, F A For the fusion of feature maps; F slow1 For the first slow path video frame set; F fast1 For the first fast path video frame set; F T STPool is the set of feature weight thresholds; σ is the average pooling along the spatiotemporal dimension; Threshold represents the threshold function; ω SS This represents the set of strong features for slow paths; ω SW ω represents the set of weak features for slow paths. FS This is the set of strong features for fast paths; ω FW The set of weak features for fast paths; ω slow For the set of slow path weights; ω fast F is the set of fast path weights; W To reconstruct the set of weak features; F S is for reconstructing a strong feature set; 3D-SENet is a 3D SENet attention mechanism operation; P is the set of reconstructed feature maps.

Images