Weakly supervised temporal action localization method and system with cross-modal enhancement and residual attention

Abstract

The application belongs to the field of computer vision, and relates to a weakly supervised temporal action localization method and system with cross-modal enhancement and residual attention. The method comprises the following steps: feature extraction; multi-head cross-modal information interaction; cross-modal attention enhancement based on RGB; residual attention enhancement based on optical flow; dual-flow feature fusion and segment-level action classification; and video-level action classification. The application has the advantages that cross-modal information is learned through information interaction of different modal features, and RGB features are enhanced by learning cross-modal information and global information within the modal, and time information is enhanced by capturing the dependency between sequence data. The enhanced features extract more task-related time and space information, so that more accurate action localization and classification results can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a weakly supervised temporal action localization method and system with cross-modal enhancement and residual attention. Background Technology

[0002] With the rapid growth of video data, how to efficiently and accurately understand and utilize video content has become an urgent problem to be solved. Research on weakly supervised temporal action localization provides an effective method for mining valuable information from large-scale video data, simplifying the data collection process by using only video-level labels for temporal action localization. Due to the lack of precise temporal boundaries, existing work on WTAL mainly follows a localization-classification pipeline. Specifically, labels are used to learn the temporal class activation sequence for each input video, representing the probability that each frame in the video belongs to each action class. Then, in the testing phase, the boundaries of action instances in the video can be located by using threshold scores on the temporal class activation sequence. To effectively distinguish action from background, many studies have improved attention mechanisms, suppressing background activation scores and highlighting action activation scores. Some works also utilize video information to generate pseudo-labels to improve the quality of class activation sequences. Although these methods have achieved significant improvements on WTAL, improving the accuracy of action boundary prediction remains a problem to be solved.

[0003] On the one hand, existing methods typically use pre-trained models to extract RGB and optical flow features for training. However, these features are not trained for the WTAL task, and the extracted feature information may not be fully utilized or may contain redundant information, resulting in inaccurate class activation sequences and thus incomplete action segment localization. On the other hand, different action instances vary in length and cover different time spans, while the optical flow and RGB features used for training are extracted in segments of 16 frames each, which cannot fully cover all action instances, making it impossible to fully utilize temporal information, which also leads to inaccurate boundary prediction. Summary of the Invention

[0004] The purpose of this invention is to address the problem of weakly supervised temporal action localization. Since the RGB and optical flow features extracted by the pre-trained model for training are not specifically designed for the WTAL task, the extracted feature information may not be fully utilized or may contain redundant information, leading to incomplete action segment localization. This invention proposes a weakly supervised temporal action localization method and system based on cross-modal enhancement and residual attention to solve the problem of insufficient utilization of effective information due to initial features not being suitable for the task. By synergistically enhancing RGB and optical flow features and effectively utilizing temporal information to generate more accurate class activation sequences, it better facilitates the localization and classification of weakly supervised temporal action localization (WTAL) tasks.

[0005] The technical solution of the present invention is as follows: In a first aspect, the present invention provides a weakly supervised temporal action localization method based on cross-modal enhancement and residual attention, comprising the following steps: S1. Feature Extraction: The pre-trained I3D network is used to extract features from the RGB and optical flow segments in the dataset to obtain RGB and optical flow features; S2. Multi-head cross-modal information interaction: The extracted RGB and optical flow features are projected to generate multi-head inputs. For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. Cross-modal attention weights are generated through softmax operation, and RGB features and optical flow features are reweighted for all heads. S3. RGB-based cross-modal attention enhancement: The reweighted RGB features are fed into the cross-modal attention module for further processing. In the cross-modal attention module, the consistency between the two modes is maintained by learning the cross-modal information of optical flow, while the global features of the RGB mode are learned to enhance the RGB features. S4. Residual attention enhancement based on optical flow: The enhanced RGB and optical flow features are fed into the LSTM residual attention module for further processing of the optical flow features, so as to fully explore the dependencies in the time series data; S5. Dual-stream feature fusion and fragment-level action classification: The enhanced RGB and optical flow features are concatenated in the channel dimension to obtain features with a channel dimension of 2048. Then, convolution operation is used to fuse the features to obtain fused features. Finally, the fused features are fed into the classification module to obtain temporal activation sequences. S6. Video-level action classification: The obtained temporal activation sequence is multiplied by the temporal attention weight to obtain the background-suppressed temporal activation sequence. The top-k aggregation strategy is used to obtain the video-level action classification probability. The action category contained in the video is predicted according to the threshold. The training process is supervised by video-level category labels.

[0006] The specific process of S2 is as follows: S21. A set of learnable weight matrices is used to calculate the interrelationships of modal characteristics of different modes. In addition, the extracted RGB and optical flow features are projected to generate multi-head inputs to obtain the extracted RGB generated multi-head input features. Generate multi-head input features from extracted optical flow features For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. The formula is as follows: in, express The dimension is ,in express Feature dimensions, Indicates the number of heads. This represents a learnable weight matrix. Indicates transpose; Then, through the correlation matrix Cross-modal attention weights for generating RGB features by performing softmax operations row by row. By transposing the correlation matrix The cross-modal attention weights of optical flow are generated by performing softmax operations row by row. Then, reweight the RGB features and optical flow features of all heads to obtain... and The formula is as follows: in, This indicates that all header RGB features are reweighted. This indicates that all head reweighted optical flow features are represented. This indicates the extracted RGB multi-head input features. This indicates that the extracted optical flow features are used to generate multi-head input features. This indicates that GELU is activated. Indicates the number of heads; Finally, all the reweighted RGB and optical flow features from the head are aggregated, and the original RGB features are recalibrated. and optical flow characteristics This results in enhanced RGB features. and optical flow characteristics The formula is as follows: ), ), in, This represents the reweighted RGB features of all headers. This represents the reweighted optical flow features across all heads; Concat indicates the splicing operation. This represents the hyperbolic tangent activation function.

[0007] The specific process for S3 is as follows: The original RGB features Average pooling and the original optical flow mode The RGB shared features are obtained by feeding them into a shared LSTM. Features shared with optical flow modes Then and The RGB features are processed separately using sigmoid operations and used as weights in the multi-head cross-modal attention module. Weighting is performed to obtain the enhanced RGB features. and the corresponding attention weights The overall formula is as follows: , , in, This represents the activation function. Indicates element-wise multiplication. It is an attention module consisting of three convolutions.

[0008] The specific process of S4 is as follows: Optical flow characteristics initially enhanced by the multi-head cross-modal attention module As input, it is fed into the first-layer LSTM module. Each LSTM module goes through a three-layer LSTM and ReLU operation, and then is added to the output of the previous layer LSTM module to obtain the result. Then perform a sigmoid operation, and then combine it with the features. Multiply the results to obtain the output of the current layer, then feed it into the next LSTM module for the same operation. Select to stack three LSTM residual blocks, and the formula for each LSTM residual block is as follows: , , , in, This represents the input to the first LSTM layer. This represents the input to the first-level residual block. This represents the output of the nth residual block. This represents the activation function. This represents the output of the nth LSTM residual block. This represents the output of the (n-1)th LSTM residual block. This represents the activation function. Indicates element-wise multiplication; Output of the last layer As weights and input features Multiplication yields the final enhanced optical flow characteristics. The output of each layer is fed into the attention module to generate attention weights. The final attention weight The formula for the weighted average is as follows: in These are the attention weights of each LSTM residual block. This indicates the attention module. This represents the weighting coefficient.

[0009] The specific formula for S5 is as follows: , in, This indicates that two features are concatenated along the channel dimension. Represents the ReLU activation function. This represents the convolution operation. Represents a time-series activation sequence. Represents a classifier. Temporal activation sequences with suppressed background.

[0010] S6 is detailed below: Classification loss The design is as follows: , in, Represents the probability of video-level classification. Indicates video-level truth labels. Representing feature dimension, Indicates the number of action classes; Furthermore, deterministic complementary categories are eliminated through video-level complementary learning loss to reduce ambiguity. The complementary learning loss function... The design is as follows: .

[0011] On the other hand, the present invention also provides a weakly supervised temporal action localization system with cross-modal enhancement and residual attention, comprising the following steps: The feature extraction unit uses a pre-trained I3D network to extract features from RGB and optical flow segments in the dataset, obtaining RGB and optical flow features; The feature enhancement unit includes a multi-head cross-modal information interaction module, an RGB-based cross-modal attention enhancement module, and an LSTM residual attention module. The multi-head cross-modal information interaction module projects the extracted RGB and optical flow features to generate multi-head inputs. For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. Cross-modal attention weights are generated through softmax operation, and RGB features and optical flow features are reweighted for all heads. The RGB-based cross-modal attention enhancement module feeds reweighted RGB features into the cross-modal attention module for further processing. In the cross-modal attention module, consistency between the two modes is maintained by learning cross-modal information of optical flow, while global features of the RGB mode are learned to enhance the RGB features. The optical flow-based residual attention enhancement module is used to feed the enhanced RGB and optical flow features into the LSTM residual attention module for further processing of the optical flow features, fully explore the dependencies in the time series data, learn richer and more complex feature representations, and enhance the optical flow features. The dual-stream feature fusion and fragment-level action classification unit has a classification module, which concatenates the obtained enhanced RGB and optical flow features in the channel dimension to obtain 2048-dimensional features in the channel dimension. The feature fusion is performed using convolution operation to obtain fused features, which are then fed into the classification module to obtain temporal activation sequences. The video-level action classification unit multiplies the obtained temporal class activation sequence with the temporal attention weight to obtain the background-suppressed temporal class activation sequence. It uses a top-k aggregation strategy to obtain the video-level action classification probability and predicts the action category contained in the video based on the threshold. The training process is supervised by video-level category labels.

[0012] Advantages and beneficial effects of the present invention; This invention enables features from two modalities to learn from each other through a multi-head cross-modal attention module, thereby uncovering correlations between different modalities. It also enhances the temporal information of optical flow features to capture temporal dependencies and fully utilize this temporal information. Furthermore, this invention enhances the cross-modal consistency of RGB features by considering both global and cross-modal local information to filter redundant RGB features and enhance useful information. The enhanced RGB and optical flow features are then used for subsequent classification to obtain class activation sequences, resulting in robust localization and classification features. Attached Figure Description

[0013] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0014] Figure 1 This is a schematic diagram of the method structure of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0016] The following provides a detailed description of a weakly supervised temporal action localization method, system, device, and storage medium for cross-modal enhancement and residual attention provided by this invention. Contents not described in detail in the embodiments of this invention belong to prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0017] Example 1 A weakly supervised temporal action localization method combining cross-modal enhancement and residual attention includes the following steps: S1. Feature Extraction: The pre-trained I3D network is used to extract features from the RGB and optical flow segments in the dataset to obtain RGB and optical flow features; S2. Multi-head cross-modal information interaction: The extracted RGB and optical flow features are projected to generate multi-head inputs. For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. Cross-modal attention weights are generated through softmax operation, and RGB features and optical flow features are reweighted for all heads. S3. RGB-based cross-modal attention enhancement: The reweighted RGB features are fed into the cross-modal attention module for further processing. In the cross-modal attention module, the consistency between the two modes is maintained by learning the cross-modal information of optical flow, while the global features of the RGB mode are learned to enhance the RGB features. S4. Residual attention enhancement based on optical flow: The enhanced RGB and optical flow features are fed into the LSTM residual attention module for further processing of the optical flow features, so as to fully explore the dependencies in the time series data; S5. Dual-stream feature fusion and fragment-level action classification: The enhanced RGB and optical flow features are concatenated in the channel dimension to obtain features with a channel dimension of 2048. Then, convolution operation is used to fuse the features to obtain fused features. Finally, the fused features are fed into the classification module to obtain temporal activation sequences. S6. Video-level action classification: The obtained temporal activation sequence is multiplied by the temporal attention weight to obtain the background-suppressed temporal activation sequence. The top-k aggregation strategy is used to obtain the video-level action classification probability. The action category contained in the video is predicted according to the threshold. The training process is supervised by video-level category labels.

[0018] The specific process of S2 is as follows: S21. A set of learnable weight matrices is used to calculate the interrelationships of modal characteristics of different modes. In addition, the extracted RGB and optical flow features are projected to generate multi-head inputs to obtain the extracted RGB generated multi-head input features. Generate multi-head input features from extracted optical flow features For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. The formula is as follows: in, express The dimension is ,in express Feature dimensions, Indicates the number of heads. This represents a learnable weight matrix. Indicates transpose; Then, through the correlation matrix Cross-modal attention weights for generating RGB features by performing softmax operations row by row. By transposing the correlation matrix The cross-modal attention weights of optical flow are generated by performing softmax operations row by row. Then, reweight the RGB features and optical flow features of all heads to obtain... and The formula is as follows: in, This indicates that all header RGB features are reweighted. This indicates that all head reweighted optical flow features are represented. This indicates the extracted RGB multi-head input features. This indicates that the extracted optical flow features are used to generate multi-head input features. This indicates that GELU is activated. Indicates the number of heads; Finally, all the reweighted RGB and optical flow features from the head are aggregated, and the original RGB features are recalibrated. and optical flow characteristics This results in enhanced RGB features. and optical flow characteristics The formula is as follows: ), ), in, This represents the reweighted RGB features of all headers. This represents the reweighted optical flow features across all heads; Concat indicates the splicing operation. This represents the hyperbolic tangent activation function.

[0019] The specific process for S3 is as follows: The original RGB features Average pooling and the original optical flow mode The RGB shared features are obtained by feeding them into a shared LSTM. Features shared with optical flow modes Then and The RGB features are processed separately using sigmoid operations and used as weights in the multi-head cross-modal attention module. Weighting is performed to obtain the enhanced RGB features. and the corresponding attention weights The overall formula is as follows: , , in, This represents the activation function. Indicates element-wise multiplication. It is an attention module consisting of three convolutions.

[0020] The specific process of S4 is as follows: Optical flow characteristics initially enhanced by the multi-head cross-modal attention module As input, it is fed into the first-layer LSTM module. Each LSTM module goes through a three-layer LSTM and ReLU operation, and then is added to the output of the previous layer LSTM module to obtain the result. Then perform a sigmoid operation, and then combine it with the features. Multiply the results to obtain the output of the current layer, then feed it into the next LSTM module for the same operation. Select to stack three LSTM residual blocks, and the formula for each LSTM residual block is as follows: , , , in, This represents the input to the first LSTM layer. This represents the input to the first-level residual block. This represents the output of the nth residual block. This represents the activation function. This represents the output of the nth LSTM residual block. This represents the output of the (n-1)th LSTM residual block. This represents the activation function. Indicates element-wise multiplication; Output of the last layer As weights and input features Multiplication yields the final enhanced optical flow characteristics. The output of each layer is fed into the attention module to generate attention weights. The final attention weight The formula for the weighted average is as follows: in These are the attention weights of each LSTM residual block. This indicates the attention module. This represents the weighting coefficient.

[0021] The specific formula for S5 is as follows: , in, This indicates that two features are concatenated along the channel dimension. Represents the ReLU activation function. This represents the convolution operation. Represents a time-series activation sequence. Represents a classifier. Temporal activation sequences with suppressed background.

[0022] S6 is detailed below: Classification loss The design is as follows: , in, Represents the probability of video-level classification. Indicates video-level truth labels. Representing feature dimension, Indicates the number of action classes; Furthermore, deterministic complementary categories are eliminated through video-level complementary learning loss to reduce ambiguity. The complementary learning loss function... The design is as follows: .

[0023] Example 2 The timing action positioning effect of this invention: During the testing phase, classes with video-level class scores below a certain threshold are first discarded. For the remaining classes, background segments are first discarded by setting a threshold on the attention weights, and then continuous portions of the remaining segments are treated as candidate action segments. After obtaining the proposals, the action confidence score q for each proposal is calculated. Finally, a soft nonmaximum suppression strategy is used to remove overlapping proposals, resulting in the final classification and localization results and confidence scores for each proposal, denoted as (c,t). s ,t e The experimental results of this invention compared with other methods on the THUMOS14 dataset are shown in Table 1, and the experimental results of this invention compared with other methods on the ActivityNet1.2 dataset are shown in Table 2.

[0024] The Chinese meanings of the various English abbreviations in the table are as follows: Where AVG(0.1:0.5) and AVG(0.1:0.7) represent the mean average precision (mAP) of tIoU from 0.1 to 0.5 and 0.1 to 0.7, respectively. Fully indicates a fully supervised method, Weakly+ indicates a weakly supervised method with additional annotations or additional training data, and Weakly indicates a weakly supervised method with only video-level label supervision.

[0025] Table 1. Comparison of the experimental results of this invention with other methods on the THUMOS14 dataset. Table 2. Comparison of experimental results of the present invention with other methods on the ActivityNet 1.2 dataset. On the THUMOS14 dataset, the present invention outperforms most methods, achieving a score of 59.5 when calculating the average mAP for tIoU from 0.1 to 0.5, and 48.9 when calculating the average mAP for tIoU from 0.1 to 0.7. On the ActivityNet1.2 dataset, the present invention outperforms most methods, achieving a score of 29.8 when calculating the average mAP for tIoU from 0.5 to 0.95.

[0026] Example 3 A weakly supervised temporal action localization system combining cross-modal enhancement and residual attention includes the following steps: The feature extraction unit uses a pre-trained I3D network to extract features from RGB and optical flow segments in the dataset, obtaining RGB and optical flow features; The feature enhancement unit includes a multi-head cross-modal information interaction module, an RGB-based cross-modal attention enhancement module, and an LSTM residual attention module. The multi-head cross-modal information interaction module projects the extracted RGB and optical flow features to generate multi-head inputs. For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. Cross-modal attention weights are generated through softmax operation, and RGB features and optical flow features are reweighted for all heads. The RGB-based cross-modal attention enhancement module feeds reweighted RGB features into the cross-modal attention module for further processing. In the cross-modal attention module, consistency between the two modes is maintained by learning cross-modal information of optical flow, while global features of the RGB mode are learned to enhance the RGB features. The optical flow-based residual attention enhancement module is used to feed the enhanced RGB and optical flow features into the LSTM residual attention module for further processing of the optical flow features, fully explore the dependencies in the time series data, learn richer and more complex feature representations, and enhance the optical flow features. The dual-stream feature fusion and fragment-level action classification unit has a classification module, which concatenates the obtained enhanced RGB and optical flow features in the channel dimension to obtain 2048-dimensional features in the channel dimension. The feature fusion is performed using convolution operation to obtain fused features, which are then fed into the classification module to obtain temporal activation sequences. The video-level action classification unit multiplies the obtained temporal class activation sequence with the temporal attention weight to obtain the background-suppressed temporal class activation sequence. It uses a top-k aggregation strategy to obtain the video-level action classification probability and predicts the action category contained in the video based on the threshold. The training process is supervised by video-level category labels.

[0027] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A weakly supervised temporal action localization method using cross-modal enhancement and residual attention, characterized in that, Includes the following steps: S1. Feature Extraction: The pre-trained I3D network is used to extract features from the RGB and optical flow segments in the dataset to obtain RGB and optical flow features; S2. Multi-head cross-modal information interaction: The extracted RGB and optical flow features are projected to generate multi-head inputs to obtain RGB and optical flow features. A learnable weight matrix is ​​used for each head and its correlation matrix is ​​calculated. Cross-modal attention weights are generated through softmax operation, and RGB and optical flow features are reweighted for all heads. S3. RGB-based cross-modal attention enhancement: The reweighted RGB features are fed into the cross-modal attention module for further processing. In the cross-modal attention module, the consistency between the two modes is maintained by learning the cross-modal information of optical flow, while the global features of the RGB mode are learned to enhance the RGB features. The specific process for S3 is as follows: The original RGB features Average pooling and the original optical flow mode The RGB shared features are obtained by feeding them into a shared LSTM. Features shared with optical flow modes Then and The RGB features are processed separately using sigmoid operations and used as weights in the multi-head cross-modal attention module. Weighting is performed to obtain the enhanced RGB features. and the corresponding attention weights The overall formula is as follows: , , in, This represents the activation function. Indicates element-wise multiplication. It is an attention module consisting of three convolutions; S4. Residual attention enhancement based on optical flow: The enhanced RGB and optical flow features are fed into the LSTM residual attention module for further processing of the optical flow features, so as to fully explore the dependencies in the time series data; S5. Dual-stream feature fusion and fragment-level action classification: The enhanced RGB and optical flow features are concatenated in the channel dimension to obtain features with a channel dimension of 2048. Then, convolution operation is used to fuse the features to obtain fused features. Finally, the fused features are fed into the classification module to obtain temporal activation sequences. S6. Video-level action classification: The obtained temporal activation sequence is multiplied by the temporal attention weight to obtain the background-suppressed temporal activation sequence. The top-k aggregation strategy is used to obtain the video-level action classification probability. The action category contained in the video is predicted according to the threshold. The training process is supervised by video-level category labels.

2. The weakly supervised temporal action localization method based on cross-modal enhancement and residual attention according to claim 1, characterized in that, The specific process of S2 is as follows: S21. A set of learnable weight matrices is used to calculate the interrelationships of modal characteristics of different modes. In addition, the extracted RGB and optical flow features are projected to generate multi-head inputs to obtain the extracted RGB generated multi-head input features. Generate multi-head input features from extracted optical flow features For each head, a learnable weight matrix is ​​used and its correlation matrix is ​​calculated. The formula is as follows: in, express The dimension is ,in express Feature dimensions, Indicates the number of heads. This represents a learnable weight matrix. Indicates transpose; Then, through the correlation matrix Cross-modal attention weights for generating RGB features by performing softmax operations row by row. By transposing the correlation matrix The cross-modal attention weights of optical flow are generated by performing softmax operations row by row. Then, reweight the RGB features and optical flow features of all heads to obtain... and The formula is as follows: in, This indicates that all header RGB features are reweighted. This indicates that all head reweighted optical flow features are represented. This indicates the extracted RGB multi-head input features. This indicates that the extracted optical flow features are used to generate multi-head input features. This indicates that GELU is activated. Indicates the number of heads; Finally, all the reweighted RGB and optical flow features from the head are aggregated, and the original RGB features are recalibrated. and optical flow characteristics This results in enhanced RGB features. and optical flow characteristics The formula is as follows: ), )， in, This is represented as the reweighted RGB features of all headers. This represents the reweighted optical flow features across all heads; Concat indicates the splicing operation. This represents the hyperbolic tangent activation function.

3. The weakly supervised temporal action localization method based on cross-modal enhancement and residual attention according to claim 1, characterized in that, The specific process of S4 is as follows: Optical flow characteristics initially enhanced by the multi-head cross-modal attention module As input, it is fed into the first-layer LSTM module. Each LSTM module goes through a three-layer LSTM and ReLU operation, and then is added to the output of the previous layer LSTM module to obtain the result. Then perform a sigmoid operation, and then combine it with the features. Multiply the results to obtain the output of the current layer, then feed it into the next LSTM module for the same operation. Select to stack three LSTM residual blocks, and the formula for each LSTM residual block is as follows: ， ， ， in, This represents the input to the first LSTM layer. This represents the input to the first-level residual block. This represents the output of the nth residual block. This represents the activation function. This represents the output of the nth LSTM residual block. This represents the output of the (n-1)th LSTM residual block. This represents the activation function. Indicates element-wise multiplication; Output of the last layer As weights and input features Multiplication yields the final enhanced optical flow characteristics. The output of each layer is fed into the attention module to generate attention weights. The final attention weight The formula for the weighted average is as follows: in These are the attention weights of each LSTM residual block. This indicates the attention module. This represents the weighting coefficient.

4. The weakly supervised temporal action localization method based on cross-modal enhancement and residual attention according to claim 1, characterized in that, The specific formula for S5 is as follows: ， in, This indicates that two features are concatenated along the channel dimension. Represents the ReLU activation function. This represents the convolution operation. Represents a time-series activation sequence. Represents a classifier. This indicates the final enhanced optical flow characteristic. This represents the enhanced RGB features. Temporal activation sequences with suppressed background.

5. The weakly supervised temporal action localization method based on cross-modal enhancement and residual attention according to claim 1, characterized in that, S6 is detailed below: Classification loss The design is as follows: ， in, Represents the probability of video-level classification. Indicates video-level truth labels. Representing feature dimension, Indicates the number of action classes; Furthermore, deterministic complementary categories are eliminated through video-level complementary learning loss to reduce ambiguity. The complementary learning loss function... The design is as follows: 。 6. A weakly supervised temporal action localization system based on cross-modal enhancement and residual attention, executing the weakly supervised temporal action localization method based on cross-modal enhancement and residual attention as described in any one of claims 1-5, characterized in that, Includes the following steps: The feature extraction unit uses a pre-trained I3D network to extract features from RGB and optical flow segments in the dataset, obtaining RGB and optical flow features; The feature enhancement unit includes a multi-head cross-modal information interaction module, an RGB-based cross-modal attention enhancement module, and an LSTM residual attention module. The multi-head cross-modal information interaction module projects the extracted RGB and optical flow features to generate multi-head inputs to obtain RGB features and optical flow features. It uses a learnable weight matrix for each head and calculates its correlation matrix. It generates cross-modal attention weights through softmax operation and reweights the RGB features and optical flow features for all heads. The RGB-based cross-modal attention enhancement module feeds reweighted RGB features into the cross-modal attention module for further processing. In the cross-modal attention module, consistency between the two modes is maintained by learning cross-modal information of optical flow, while global features of the RGB mode are learned to enhance the RGB features. The optical flow-based residual attention enhancement module is used to feed the enhanced RGB and optical flow features into the LSTM residual attention module for further processing of the optical flow features, fully explore the dependencies in the time series data, learn richer and more complex feature representations, and enhance the optical flow features. The dual-stream feature fusion and fragment-level action classification unit has a classification module, which concatenates the obtained enhanced RGB and optical flow features in the channel dimension to obtain 2048-dimensional features in the channel dimension. The feature fusion is performed using convolution operation to obtain fused features, which are then fed into the classification module to obtain temporal activation sequences. The video-level action classification unit multiplies the obtained temporal class activation sequence with the temporal attention weight to obtain the background-suppressed temporal class activation sequence. It uses a top-k aggregation strategy to obtain the video-level action classification probability and predicts the action category contained in the video based on the threshold. The training process is supervised by video-level category labels.

Images