Weak supervision time sequence action positioning method based on progressive action perception

Abstract

The invention relates to a weak supervision time sequence action positioning method based on progressive action perception, and belongs to the field of computer vision. The method comprises the following steps: extracting double-flow features from a video by using a pre-trained I3D network, and performing data enhancement through a DDG-Net network to obtain double-flow enhanced features; inputting the enhanced features into a residual convolution module to form double-flow fusion features; performing frame level enhancement on results of the previous two steps by combining an inter-frame difference and a double-branch attention mechanism to obtain double-flow optimization features; a global information matrix is constructed based on the optimized features, the features are further optimized, enhanced double-flow optimized features are obtained through fusion, and then attention mechanism processing is carried out; a class activation graph is generated according to the optimization features, an average attention score is calculated, an attention-optimized class activation graph is obtained through matrix multiplication, and action classification and positioning are carried out; and carrying out training optimization on the process through a total loss function. According to the invention, the accuracy of action positioning and classification can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a weakly supervised temporal action localization method based on progressive action perception. Background Art

[0002] With the rapid growth of video data, intelligent video processing has become a key research direction. Temporal action localization is a challenging task in video analysis, aiming to accurately locate the start and end times of specific actions in untrimmed videos. Unlike traditional action recognition, temporal action localization requires not only identifying action categories but also accurately determining temporal boundaries, which is crucial for applications such as video understanding, retrieval, and surveillance. Recently, fully supervised temporal action localization has made progress, but its scalability and practical application are limited by its reliance on expensive frame-level annotations. To address this issue, weakly supervised temporal action localization has attracted attention. It relies solely on video-level labels, avoiding the high cost of frame-level annotation. Many weakly supervised temporal action localization methods have demonstrated promising results. Among them, multi-instance learning-based methods divide the video into multiple segments and generate a temporal class activation map (T-CAM) to estimate whether each segment contains the target action. Top-k pooling is then used to select high-scoring segments for subsequent localization. Attention-based methods, on the other hand, dynamically adjust weights to focus on important segments, improving the accuracy of foreground action recognition. Although existing weakly supervised temporal action localization methods have made progress, challenges still exist: many models over-rely on long-range information and ignore local details; or focus too much on short-term actions, resulting in interference from redundant information and affecting accurate positioning. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a weakly supervised temporal action localization method based on progressive action perception.

[0004] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: The present invention provides a method for weakly supervised temporal action localization based on progressive action perception, comprising the following steps: S1. Obtain the video for action localization and extract features from it using the pre-trained I3D network to obtain dual-stream video features. Then, use the DDG-Net network as the base network to perform data enhancement on the dual-stream video features to obtain dual-stream data enhancement features. S2. The data enhancement features of the two streams are passed through the residual convolution module to capture short action features, resulting in the fusion features of the two streams. S3. Enhance the results of steps S1 and S2 at the feature frame level through inter-frame difference and dual-branch attention mechanism to obtain the optimized features of the dual stream; S4. Construct a global information matrix based on the dual-stream optimization features. Further optimize the dual-stream optimization features according to the global information matrix to obtain the dual-stream features optimized by the global information matrix. Add the dual-stream optimization features and the dual-stream features optimized by the global information matrix element by element to obtain the enhanced dual-stream optimization features. Process the enhanced dual-stream optimization features through the attention mechanism to obtain the attention score. After performing a convolution operation on the enhanced dual-stream optimization features, use the attention mechanism to mine the attention of the features and obtain the mean attention score. S5. Obtain a class activation map based on the enhanced dual-stream optimization features, calculate the average attention score based on the attention score and the mean of the attention scores obtained in step S4, perform matrix multiplication on the class activation map and the average attention score to obtain an attention-optimized class activation map, and then obtain the action classification and localization results; S6. During the training from step S1 to step S5, training optimization is performed using the total loss function.

[0005] Furthermore, step S1 specifically includes: Use the I3D network pre-trained on the Kinetics dataset to perform deep feature analysis on the RGB images and optical flow images in the video, capture the temporal and spatial information in the video, and obtain RGB flow features and optical flow modal features ; Using DDG-Net network as the basic network to analyze RGB stream features and optical flow modal features Perform data enhancement to obtain RGB stream data enhancement features and optical flow modality data enhancement features .

[0006] Furthermore, step S2 specifically includes: Enhance the features of RGB stream data and optical flow modality data enhancement features The input is sent to the residual convolution module, and after being processed by two residual convolution blocks, the element-by-element summation operation is performed with the original features to obtain the RGB stream fusion feature. and optical flow modality fusion features , the formula is as follows: , , Among them, B represents the batch size, T represents the time dimension, D represents the feature channel dimension, ConvDown represents downsampling convolution, Conv represents convolution operation, and ConvUp represents upsampling convolution. Represents the Relu activation function.

[0007] Furthermore, step S3 specifically includes: RGB stream features , optical flow modal features , RGB stream fusion features And optical flow modality fusion features Input into the attention mechanism to obtain the corresponding attention score, the formula is as follows: , , Among them, Attention represents the attention mechanism; 、 、 、 Represents RGB stream features respectively , optical flow modal features , RGB stream fusion features , optical flow modality fusion features The corresponding attention score; The weighted mean is calculated based on the obtained attention scores, and the frame difference is obtained by calculating the difference between frames. The RGB stream fusion features are respectively calculated by combining the weighted mean and the frame difference. and optical flow modality fusion features Perform weighted summation to obtain RGB flow optimization features , the formula is as follows: in, Express Find the mean in T dimension, Express Find the mean in T dimension, Indicates the weighted mean of the RGB stream without combining the frame difference, Represents the weighted mean of the RGB stream combined with the frame difference, Represents the set of frame differences of the RGB stream in the time dimension T; Indicates the score of frame 0; Represents the frame difference between each two adjacent frames of the RGB stream, subtracting the next frame from the previous frame; Represents the irth frame information of the RGB stream; Indicates the ir+1th frame information of the RGB stream; Represents the matrix multiplication operation; similarly, the weighted mean of the optical flow combined with the frame difference is obtained and optical flow modality optimization features .

[0008] Furthermore, step S4 specifically includes: Optimizing features for RGB stream , optimize the RGB flow features And its transpose through matrix multiplication and then through the Softmax activation function to obtain the global information matrix M; according to the global information matrix M and RGB flow optimization features , get the RGB flow optimization feature of global information matrix optimization ; Optimize the RGB stream features and RGB flow optimization features optimized by global information matrix Enhanced RGB stream optimization features are obtained after pixel-by-pixel addition , enhanced RGB flow optimization features After the attention mechanism is calculated, the enhanced RGB flow optimization features are obtained The attention score is expressed as follows: , , , , in, represents the Softmax activation function, Represents RGB stream optimization features The transpose of represents the L2 norm, represents the identity matrix, Represents enhanced RGB stream optimization features Attention score; similarly, the optical flow modality optimizes the feature After the above operations, the enhanced optical flow modality optimization features are obtained Attention score .

[0009] Enhanced RGB stream optimization features The convolution operation is used to extract long-term action information, and the attention mechanism is used to mine the attention of features to obtain enhanced RGB stream optimization features. The mean attention score , the formula is as follows: , , , in, Represents the enhanced RGB flow optimization features after convolution The corresponding first attention score, Represents the enhanced RGB flow optimization features after convolution The corresponding second attention score, represents the sigmoid activation function, represents the element-wise multiplication operation, Represents enhanced RGB stream optimization features The mean of the attention scores; similarly, the enhanced optical flow modality optimization features The enhanced optical flow modality optimization features obtained through the above operations The mean attention score .

[0010] Furthermore, step S5 specifically includes: Enhanced RGB stream optimization features and enhanced optical flow modality optimization features Splicing in the channel dimension to obtain splicing features , splicing features After processing by the classifier, the class activation map is obtained ; Optimize features based on enhanced RGB stream Attention score , Enhanced RGB stream optimization features The mean attention score , enhanced optical flow modality optimization features Attention score And enhanced optical flow modality optimization features The mean attention score Calculate the average attention score of RGB flow and optical flow , the class activation map and the average attention score Perform matrix multiplication to obtain the attention-optimized class activation map , the formula is as follows: Among them, C represents the total number of categories that may exist in the video; C+1 represents the score including background; Represents a splicing operation; Represents a classifier; sum represents cumulative calculation; the class activation map of the attention optimization and the average attention score Input into the temporal action positioning module for temporal action positioning to obtain action classification and positioning results.

[0011] Furthermore, step S6 specifically includes: The total loss function includes collaborative attention loss, receptive field consistency loss and weight alignment loss; The collaborative attention loss The formula is expressed as: ; The receptive field consistency loss The formula is expressed as: ; The weight alignment loss The formula is expressed as ;in, Represents the mean square error; the final total loss function It can be expressed as: ;in, Represents the loss in the DDG-Net network.

[0012] The advantages of the present invention are: The present invention uses a pre-trained I3D model to extract RGB stream and optical flow modal features in the video to obtain dual-stream video features, and adopts a two-layer residual convolution structure to capture short-term action features, effectively reducing computational costs and background interference; enhances frame-level features through inter-frame differences and a dual-branch attention mechanism, improves action representation and fine-tunes action boundaries; in order to enhance the perception of long-term actions, the global context information is combined to construct a global information matrix using matrix multiplication, and long-term dependencies are captured through convolution superposition operations; in the modality fusion stage, different attention mechanisms and convolution operations are introduced to accurately fuse features of short-term and long-term action representations; finally, model training is optimized through collaborative attention loss, receptive field consistency loss and weight alignment loss, and background-suppressed temporal class activation maps (T-CAMs) are generated using temporal class activation maps (T-CAMs) and attention scores during inference. Combined with Soft-NMS to suppress redundant candidate segments, accurate action localization and classification can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0014] Figure 1 is a flow chart of the steps of the method of the present invention; Figure 2 Comparison of positioning video moments between the method of the present invention and the existing method; Figure 3 Comparison of the positioning video peaks between the method of the present invention and the existing method. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments derived by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] Example 1 In this embodiment, Figure 1 As shown, the present invention provides a weakly supervised temporal action localization method based on progressive action perception, and the specific steps include: S1. Obtain the video for action localization and use the pre-trained I3D network to extract features from the video to obtain dual-stream video features. Use the DDG-Net network as the base network to perform data enhancement on the dual-stream video features to obtain dual-stream data enhancement features.

[0017] Specifically, the I3D network pre-trained on the Kinetics dataset is used to perform deep feature analysis on the RGB images and optical flow images in the video, capturing the temporal and spatial information in the video and obtaining the RGB flow features. and optical flow modal features ; Using DDG-Net network as the basic network to analyze RGB stream features and optical flow modal features Perform data enhancement to obtain RGB stream data enhancement features and optical flow modality data enhancement features ; to further enrich the feature representation. This data augmentation method significantly improves the performance and generalization ability of the model in the action localization task.

[0018] S2. The data enhancement features of the dual streams are passed through the residual convolution module to capture short action features and obtain the fusion features of the dual streams.

[0019] Specifically, the RGB stream data is enhanced and optical flow modality data enhancement features The input is sent to the residual convolution module, and after being processed by two residual convolution blocks, the element-by-element summation operation is performed with the original features to obtain the RGB stream fusion feature. and optical flow modality fusion features , the formula is as follows: , , Among them, B represents the batch size, T represents the time dimension, D represents the feature channel dimension, ConvDown represents downsampling convolution, which is used to adjust the channel dimension of the feature from D dimension to D / 2 dimension by using a 1×1 convolution kernel, Conv represents convolution operation, which is used to further extract features on the channel dimension of D / 2 by using a 3×3 convolution kernel, and ConvUp represents upsampling convolution, which is used to adjust the feature from D / 2 dimension to D dimension by using a 1×1 convolution kernel. Represents the Relu activation function.

[0020] S3. Enhance the results of steps S1 and S2 at the feature frame level through inter-frame difference and dual-branch attention mechanism to obtain the optimized features of the dual stream.

[0021] Specifically, the RGB stream features , optical flow modal features , RGB stream fusion features And optical flow modality fusion features Input into the attention mechanism, further optimize the action representation and fine-tune the action boundary to obtain the corresponding attention score, which is expressed as follows: , , Among them, Attention represents the attention mechanism; 、 、 、 Represents RGB stream features respectively , optical flow modal features , RGB stream fusion features , optical flow modality fusion features The corresponding attention score; The weighted mean is calculated based on the obtained attention scores, and the frame difference is obtained by calculating the difference between frames. The RGB stream fusion features are respectively calculated by combining the weighted mean and the frame difference. and optical flow modality fusion features Perform weighted summation to obtain RGB flow optimization features , the formula is as follows: in, Express Find the mean in T dimension, Express Find the mean in T dimension, Represents the weighted mean of the RGB stream without combining the frame difference; Represents the weighted mean of the RGB stream combined with the frame difference, which is used for the subsequent loss value calculation; Represents the set of frame differences of the RGB stream in the time dimension T; Indicates the score of the 0th frame (because the time dimension of the video is T, calculating the frame difference between each two will get the score of the T-1 frame, so the first value of the frame difference is filled with 0); Represents the frame difference between each two adjacent frames of the RGB stream, subtracting the next frame from the previous frame; Represents the irth frame information of the RGB stream; Indicates the ir+1th frame information of the RGB stream; Represents the matrix multiplication operation; similarly, the weighted mean of the optical flow combined with the frame difference is obtained and optical flow modality optimization features .

[0022] S4. Construct a global information matrix based on the dual-stream optimization features. Further optimize the dual-stream optimization features according to the global information matrix to obtain the dual-stream features optimized by the global information matrix. Add the dual-stream optimization features and the dual-stream features optimized by the global information matrix element by element to obtain the enhanced dual-stream optimization features. Process the enhanced dual-stream optimization features through the attention mechanism to obtain the attention score. After performing a convolution operation on the enhanced dual-stream optimization features, use the attention mechanism to mine the attention of the features and obtain the mean attention score. Specifically, in order to enhance the perception of long-term actions, the global context information is combined and matrix multiplication is used to construct a global information matrix. By calculating the similarity between features, the model can identify frames with strong correlations, thereby capturing long-term dependencies. Through convolutional stacking operations, the model gradually expands the receptive field, further enhancing the perception of long-term actions. For RGB stream optimization features , optimize the RGB flow features And its transpose through matrix multiplication and then through the Softmax activation function to obtain the global information matrix M; according to the global information matrix M and RGB flow optimization features , get the RGB flow optimization feature of global information matrix optimization ; Optimize the RGB stream features and RGB flow optimization features optimized by global information matrix Enhanced RGB stream optimization features are obtained after pixel-by-pixel addition , enhanced RGB flow optimization features After the attention mechanism is calculated, the enhanced RGB flow optimization features are obtained The attention score is expressed as follows: , , , , in, represents the Softmax activation function, Represents RGB stream optimization features The transpose of represents the L2 norm, represents the identity matrix, Represents enhanced RGB stream optimization features Attention score; similarly, the optical flow modality optimizes the feature After the above operations, the enhanced optical flow modality optimization features are obtained Attention score .

[0023] Specifically, the enhanced RGB flow optimization features The convolution operation is used to extract long action information, and the attention mechanism is used to mine the attention of features to obtain enhanced RGB stream optimization features. The mean attention score , the formula is as follows: , , , in, Represents the enhanced RGB flow optimization features after convolution The corresponding first attention score, Represents the enhanced RGB flow optimization features after convolution The corresponding second attention score, represents the sigmoid activation function, represents the element-wise multiplication operation, Represents enhanced RGB stream optimization features The mean of the attention scores; similarly, the enhanced optical flow modality optimization features The enhanced optical flow modality optimization features obtained through the above operations The mean attention score .

[0024] S5. Obtain a class activation map based on the enhanced dual-stream optimization features, calculate the average attention score based on the attention score and the mean of the attention scores obtained in step S4, perform matrix multiplication on the class activation map and the average attention score to obtain an attention-optimized class activation map, and then obtain the action classification and positioning results.

[0025] Specifically, the enhanced RGB flow optimization features and enhanced optical flow modality optimization features Splicing in the channel dimension to obtain splicing features , splicing features After processing by the classifier, the class activation map is obtained ; Optimize features based on enhanced RGB stream Attention score , Enhanced RGB stream optimization features The mean attention score , enhanced optical flow modality optimization features Attention score And enhanced optical flow modality optimization features The mean attention score Calculate the average attention score of RGB flow and optical flow , the class activation map and the average attention score Perform matrix multiplication to obtain the attention-optimized class activation map , the formula is as follows: Among them, C represents the total number of categories that may exist in the video; C+1 represents the score including background; Represents a splicing operation; Represents a classifier; sum represents cumulative calculation; the class activation map is a time series score used to represent the correlation between each time point in the video and each action category in temporal action positioning, and is used to locate the time period when the action occurs. and the average attention score Input into the temporal action positioning module to carry out the temporal action positioning process and obtain the action classification and positioning results.

[0026] The specific process of the sequential action positioning module is as follows: The specific positioning information obtained is divided into four items: [Ts, Te, ,θ], where Ts and Te represent the start and end of the action respectively, represents the possible action categories in the video, θ represents the credibility score, and the process of obtaining the four pieces of positioning information is as follows: (1) Based on the obtained average attention score A, a thresholding operation is performed to determine the start time Ts and end time Te of the action. The action threshold is set to 0.8, where the value range of the attention score A is between [0, 1]. For a video of length T, if the attention scores of several consecutive frames of the video are greater than 0.8, until an action frame with an attention score less than 0.8 is encountered, this continuous frame is considered to be an action segment. The start frame and end frame of the segment are marked as the start time Ts and end time Te of the action respectively. The thresholding operation is performed on all videos in batch B to extract the start time and end time of all actions in each video; (2) Definition Indicates the type of action, +1 indicates the category containing background information, based on the class activation map optimized by attention Perform Top-k operation on T dimension, and then get K×( +1) dimension vector, averaged over K dimensions to obtain 1×( +1), the vector dimension obtained after removing the background class through vector clipping operation is 1× The vector is then classified and predicted by the SoftMax activation function, and is selected according to the preset threshold (usually 0.2). If the 1× The value in is greater than the preset threshold (usually 0.2), then the corresponding 1× The value of is set to 1, otherwise it is 0, so we get 1× It represents the Label vector containing the category. If there are multiple values ​​1 in the Label vector, it is considered that this video may contain several action categories, and The number can be greater than or equal to 1.

[0027] (3) For the calculation of the credibility score θ, follow the following formula: , , , in, represents the credibility score, represents the intra-class credibility score, represents the out-of-class credibility score, Indicates an adjustable parameter (usually 0.2), Represents the class activation map after attention suppression Take all the time information from Ts to Te, Lm represents the left and right extension parameters, and the calculation method is Lm= (Te-Ts) / 4, Represents the class activation map after attention suppression Winning To all the time information in Te, Represents the class activation map after attention suppression Winning All time information from Te+Lm.

[0028] After the above three steps, we obtain the information [Ts, Te, C, θ], which contains the start time Ts and end time Te of the video, the possible action categories C in the video, and the credibility score θ. At this point, the entire process of action localization and classification has been completed in the task of weakly supervised temporal action localization.

[0029] S6. During the training of steps S1 to S5, the training is optimized using the total loss function. Specifically, the total loss function includes collaborative attention loss, receptive field consistency loss and weight alignment loss; The collaborative attention loss The formula is expressed as: ; The receptive field consistency loss The formula is expressed as: ; The weight alignment loss The formula is expressed as ;in, Represents the mean square error; the final total loss function It can be expressed as: ;in, Represents the loss in the DDG-Net network.

[0030] Collaborative attention loss aligns the attention distribution of dual-stream features, ensuring that the model can extract consistent action information from different modalities. Receptive field consistency loss constrains the receptive fields of different modules to ensure model consistency at different scales. Weight alignment loss optimizes the weight distribution during localization, increasing the model's focus on key action areas.

[0031] Example 2 This paper uses a pre-trained I3D model for the THUMOS14 dataset (an action dataset) and also extracts features from the ActivityNet1.2 and ActivityNet1.3 datasets (video datasets). All experiments were run on an A100 RTX graphics card with 40GB of video memory. During the testing phase, categories with video-level category scores below a certain threshold were first discarded. For the remaining categories, the paper discards background segments by setting an attention weight threshold. The contiguous portions of the remaining segments are then used as candidate action segments. After proposals are generated, an action confidence score is calculated for each proposal. Next, a non-maximum suppression (Soft-NMS) strategy is employed to remove overlapping proposals. Finally, the classification, localization, and confidence scores for each proposal are obtained. In the experimental section, the experimental results of this paper compared with other methods on the THUMOS14 dataset are shown in Table 1, on the ActivityNet1.2 dataset in Table 2, and on the ActivityNet1.3 dataset in Table 3. In the table, Avg(0.1:0.5), Avg(0.1:0.7), and Avg(0.5:0.95) represent the mean average precision (mAP) for temporal intersection-over-union (tIoU) ranges of 0.1–0.5, 0.1–0.7, and 0.5–0.95, respectively. The proposed method was compared with other methods on the THUMOS14, ActivityNet1.2, and ActivityNet1.3 datasets. The experimental results are shown in Tables 1, 2, and 3.

[0032] The comparison of the experimental results of the present invention with other methods on the THUMOS14 dataset is shown in Table 1, where 0.1 to 0.7 represents the mean accuracy of each item under the intersection-union ratio of 0.1 to 0.7, Avg(0.1:0.5) represents the average accuracy under the intersection-union ratio of 0.1 to 0.5, and Avg(0.1:0.7) represents the average accuracy under the intersection-union ratio of 0.1 to 0.7; Table 1 Comparison of the experimental results of the present invention and other methods on the THUMOS14 dataset The TAL-Net model is a rethinking of the Fast RCNN architecture model, the P-MIL model is a multi-instance learning model based on the proposal, the PivoTAL model is a priori-driven supervision model, the DGCNN model is a dynamic graph modeling model, the DELU model is a dual-evidence learning model, the IGLK model is a foreground enhancement model integrating global and local knowledge, the ISSF model is a model for inferring significant fragment features, the EDE model is a model for efficiently eliminating double confusion, and the baseline model is a discriminant-driven graph network model. All of the above models are open source models on the Internet. At the same time, in order to avoid the table being too long, this invention uses English abbreviations.

[0033] The comparison of the experimental results of the present invention with other methods on the ActivityNet1.2 dataset is shown in Table 2, where 0.5 to 0.95 represents the mean accuracy of each item under the intersection-union ratio of 0.5 to 0.95, and Avg(0.5:0.95) represents the average accuracy under the intersection-union ratio of 0.5 to 0.95; Table 2 Comparison of the experimental results of the present invention and other methods on the ActivityNet1.2 dataset The EPNet model is an integrated prototype network model, and the baseline model is a discriminant-driven graph network model. All of these models are open source models on the Internet.

[0034] The comparison of the experimental results of the present invention with other methods on the ActivityNet1.3 dataset is shown in Table 3, where 0.5 to 0.95 represents the mean accuracy of each item when the intersection-union ratio is 0.5 to 0.95, and Avg(0.5:0.95) represents the average accuracy of the total when the intersection-union ratio is 0.5 to 0.95; Table 3 Comparison of the experimental results of the present invention and other methods on the ActivityNet1.3 dataset As can be seen from Tables 1, 2, and 3, on the THUMOS14 dataset, the method of the present invention, as a weakly supervised method that relies only on video-level label supervision, performs significantly better than similar methods. When the tIoU value is between 0.1 and 0.7, the average mAP reaches 49.6%. Compared with fully supervised methods and weakly supervised methods that rely on additional annotations or training data, the method of the present invention shows significant advantages in positioning accuracy. On the ActivityNet1.2 dataset, the present invention also demonstrates superior performance, especially when the tIoU value ranges from 0.5 to 0.95, the average mAP reaches 27.4%. In addition, on the ActivityNet1.3 dataset, the performance of the method of the present invention also exceeds that of most existing methods, especially when the tIoU value is between 0.5 and 0.95, the average mAP is 27.3%. These results show that the positioning effect of the method of the present invention on multiple datasets is significantly better than that of traditional methods, and it has strong practicality and promotion value.

[0035] Example 3 In order to highlight the advantages of the method of the present invention, the technical solution of the present invention is visually compared with the technical solution of the benchmark model. The method of the present invention adopts the discriminant driven graph model as the benchmark model, such as Figure 2 and Figure 3 The dotted box in the figure represents the true value information, where the first row shows the actual result, the second row shows the visualization effect of the benchmark model, and the third row shows the visualization effect of the method of the present invention. The method of the present invention is significantly better than the benchmark model in positioning accuracy. Figure 2 The visualization results show that the visualization results of the proposed model are more accurate in locating the start and end times of the video than those of the benchmark model. In contrast, the positioning results of the benchmark model are not very accurate in locating the start and end times of the video, with some cases of delayed start or early end. Figure 3 The visualization results show that the video peaks corresponding to the visualization results of the proposed model are more obvious than those of the baseline model. In contrast, the baseline model's ability to capture video peaks is poor, and in some cases, it misses key action segments. This further verifies the advantages of the proposed method in action localization, especially in weakly supervised scenarios, where it can effectively improve the accuracy and precision of localization.

[0036] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A weakly supervised temporal action localization method based on progressive action perception, characterized in that: The following steps are involved: S1. Obtain the video for action localization and extract features from the video using the pre-trained I3D network to obtain dual-stream video features; The DDG-Net network is used as the basic network to perform data enhancement on the dual-stream video features to obtain the dual-stream data enhancement features; S2. The data enhancement features of the two streams are passed through the residual convolution module to capture short action features, resulting in the fusion features of the two streams. S3. Enhance the results of steps S1 and S2 at the feature frame level through inter-frame difference and dual-branch attention mechanism to obtain the optimized features of the dual stream; S4. Construct a global information matrix based on the dual-stream optimization features. Further optimize the dual-stream optimization features according to the global information matrix to obtain the dual-stream features optimized by the global information matrix. Add the dual-stream optimization features and the dual-stream features optimized by the global information matrix element by element to obtain the enhanced dual-stream optimization features. Process the enhanced dual-stream optimization features through the attention mechanism to obtain the attention score. After performing a convolution operation on the enhanced dual-stream optimization features, use the attention mechanism to mine the attention of the features and obtain the mean attention score. S5. Obtain a class activation map based on the enhanced dual-stream optimization features, calculate the average attention score based on the attention score and the mean of the attention scores obtained in step S4, perform matrix multiplication on the class activation map and the average attention score to obtain an attention-optimized class activation map, and then obtain the action classification and localization results; S6. During the training from step S1 to step S5, training optimization is performed using the total loss function.

2. The method for weakly supervised temporal action localization based on progressive action perception according to claim 1 is characterized in that: Step S1 specifically includes: Use the I3D network pre-trained on the Kinetics dataset to perform deep feature analysis on the RGB images and optical flow images in the video, capture the temporal and spatial information in the video, and obtain RGB flow features and optical flow modal features ; Using DDG-Net network as the basic network to analyze RGB stream features and optical flow modal features Perform data enhancement to obtain RGB stream data enhancement features and optical flow modality data enhancement features .

3. The method for weakly supervised temporal action localization based on progressive action perception according to claim 2 is characterized in that: Step S2 specifically includes: Enhance the features of RGB stream data and optical flow modality data enhancement features The input is sent to the residual convolution module, and after being processed by two residual convolution blocks, the element-by-element summation operation is performed with the original features to obtain the RGB stream fusion feature. and optical flow modality fusion features , the formula is as follows: ， ， Among them, B represents the batch size, T represents the time dimension, D represents the feature channel dimension, ConvDown represents downsampling convolution, Conv represents convolution operation, and ConvUp represents upsampling convolution. Represents the Relu activation function.

4. The method for weakly supervised temporal action localization based on progressive action perception according to claim 3 is characterized in that: Step S3 specifically includes: RGB stream features , optical flow modal features , RGB stream fusion features And optical flow modality fusion features Input into the attention mechanism to obtain the corresponding attention score, the formula is as follows: ， ， Among them, Attention represents the attention mechanism; 、 、 、 Represents RGB stream features respectively , optical flow modal features , RGB stream fusion features , optical flow modality fusion features The corresponding attention score; The weighted mean is calculated based on the obtained attention scores, and the frame difference is obtained by calculating the difference between frames. The RGB stream fusion features are respectively calculated by combining the weighted mean and the frame difference. and optical flow modality fusion features Perform weighted summation to obtain RGB flow optimization features , the formula is as follows: in, Express Find the mean in T dimension, Express Find the mean in T dimension, Indicates the weighted mean of the RGB stream without combining the frame difference, Represents the weighted mean of the RGB stream combined with the frame difference, Represents the set of frame differences of the RGB stream in the time dimension T; Indicates the score of frame 0; Represents the frame difference between each two adjacent frames of the RGB stream, subtracting the next frame from the previous frame; Represents the irth frame information of the RGB stream; Indicates the ir+1th frame information of the RGB stream; Represents the matrix multiplication operation; similarly, the weighted mean of the optical flow combined with the frame difference is obtained and optical flow modality optimization features .

5. The method for weakly supervised temporal action localization based on progressive action perception according to claim 4 is characterized in that: Step S4 specifically includes: Optimizing features for RGB stream , optimize the RGB flow features And its transpose through matrix multiplication and then through the Softmax activation function to obtain the global information matrix M; according to the global information matrix M and RGB flow optimization features , get the RGB flow optimization feature of global information matrix optimization ; Optimize the RGB stream features and RGB flow optimization features optimized by global information matrix Enhanced RGB stream optimization features are obtained after pixel-by-pixel addition , enhanced RGB flow optimization features After the attention mechanism is calculated, the enhanced RGB flow optimization features are obtained The attention score is expressed as follows: , , ， , in, represents the Softmax activation function, Represents RGB stream optimization features The transpose of represents the L2 norm, represents the identity matrix, Represents enhanced RGB stream optimization features Attention score; similarly, the optical flow modality optimizes the feature After the above operations, the enhanced optical flow modality optimization features are obtained Attention score .

6. The method for weakly supervised temporal action localization based on progressive action perception according to claim 5, characterized in that: Step S4 further includes: Enhanced RGB stream optimization features The convolution operation is used to extract long-term action information, and the attention mechanism is used to mine the attention of features to obtain enhanced RGB stream optimization features. The mean attention score , the formula is as follows: ， ， ， in, Represents the enhanced RGB stream optimization features after convolution The corresponding first attention score, Represents the enhanced RGB stream optimization features after convolution The corresponding second attention score, represents the sigmoid activation function, represents the element-wise multiplication operation, Represents enhanced RGB stream optimization features The mean of the attention scores; similarly, the enhanced optical flow modality optimization features The enhanced optical flow modality optimization features obtained through the above operations The mean attention score .

7. The method for weakly supervised temporal action localization based on progressive action perception according to claim 6, characterized in that: Step S5 specifically includes: Enhanced RGB stream optimization features and enhanced optical flow modality optimization features Splicing in the channel dimension to obtain splicing features , splicing features After processing by the classifier, the class activation map is obtained ; Optimize features based on enhanced RGB stream Attention score , Enhanced RGB stream optimization features The mean attention score , enhanced optical flow modality optimization features Attention score And enhanced optical flow modality optimization features The mean attention score Calculate the average attention score of RGB flow and optical flow , the class activation map and the average attention score Perform matrix multiplication to obtain the attention-optimized class activation map , the formula is as follows: Where C represents the total number of possible categories in the video; C+1 represents the score including the background; Represents a splicing operation; Represents a classifier; sum represents cumulative calculation; the class activation map of the attention optimization and the average attention score Input into the temporal action positioning module for temporal action positioning to obtain action classification and positioning results.

8. The method for weakly supervised temporal action localization based on progressive action perception according to claim 7, characterized in that: Step S6 specifically includes: The total loss function includes collaborative attention loss, receptive field consistency loss and weight alignment loss; The collaborative attention loss The formula is expressed as: ; The receptive field consistency loss The formula is expressed as: ; The weight alignment loss The formula is expressed as ;in, Represents the mean square error; the final total loss function It can be expressed as: ;in, Represents the loss in the DDG-Net network.

Images