A Video Temporal Action Localization Method Based on Semantic-Level Temporal Association Modeling

Abstract

The present invention relates to the field of computer vision, and discloses a video temporal action localization method based on semantic-level temporal correlation modeling. First, a foreground mask generation module is designed to adaptively generate a foreground mask, which is used in the subsequent process to guide the Transformer module to model the semantic-level correlation with action frames. The mask-guided Transformer module uses the foreground mask to guide the attention mechanism to learn semantic-level similarities. Finally, the two modules are fused in a unified framework to implement an end-to-end deep neural network for temporal action localization. The present invention can fully model the semantic-level temporal correlation between video frames and action segments, capture the feature information of actions and suppress the interference of background noise, so that the temporal action localization result has better accuracy and can accurately locate the action interval in complex videos.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for localizing video temporal actions based on semantic-level temporal association modeling. Background Art

[0002] Video temporal action localization is one of the basic tasks in the field of computer vision. Its task is to locate the start time, end time, and corresponding category of human actions in long videos. It has important application value and significance in real life, such as video retrieval, video recommendation, etc. Driven by deep learning, the field of intelligent video analysis has flourished. The present invention is mainly aimed at the task of video temporal action localization based on deep learning, and automatically detects the temporal intervals of human actions in videos based on deep learning algorithms. Performing temporal action localization on videos and realizing automated processing of video data have broad application prospects in the fields of intelligent monitoring, virtual reality, robot control, etc., and are of great significance for promoting the development of intelligent video analysis technology.

[0003] The key task of temporal action localization is modeling the temporal relationships between different timestamps. Current methods can be categorized into three types based on the scale at which temporal relationships are modeled: local-level, proposal-level, and global-level methods. Local-level methods include anchor-box-based regression and frame-level matching. Anchor-box-based regression methods uniformly distribute predefined anchor boxes of varying sizes across a video segment and regress action segments by mining the contextual information of the video segments within the anchor boxes. Limited by the size of the anchor boxes, these methods are unable to generate proposal segments with greater temporal flexibility. Frame-level matching methods first estimate the probability of each frame being the start and end of an action and then match the start and end frames to generate action segments. These methods can predict localized segments with greater temporal scale flexibility. However, local-level methods only exploit connections between adjacent frames in the video, making action boundary probabilities highly sensitive to noise in the video and leading to incorrect boundary delineation. Proposal-level methods construct feature representations of proposals by sampling video features at different timestamps within each proposal to capture more temporal context. They utilize two-dimensional convolutional networks on proposal feature maps or employ graph convolutional networks to model relationships between proposals. Global-level methods adopt a “query and retrieval” process or utilize visual Transformers to encode global temporal dependencies.

[0004] However, in real-world scenarios, a video containing an action sequence often includes some background frames that are irrelevant to the action. These frames often provide no effective information for video understanding and may even interfere with the video. Existing methods cannot effectively prevent the influence of these background frames on the localization of the action sequence, and are unable to avoid the interference of background frames. Summary of the Invention

[0005] To address the above problems, the present invention proposes a video temporal action localization method based on semantic-level temporal correlation modeling. The present invention studies the semantic-level correlation between foreground (action segments) and background (non-action frames) in videos from two perspectives: 1) Intra-semantic similarity: For foreground segments, learning intra-semantic similarity from other action segments can extract more supportive clues. 2) Inter-semantic similarity: For background segments, capturing the inter-semantic distance between them and the foreground helps distinguish their similar appearance or motion patterns from actions, thereby suppressing false positive detections and background interference.

[0006] To achieve the above object, the present invention provides a video temporal action localization method based on semantic-level temporal correlation modeling, comprising the following steps:

[0007] S1. Use a video feature extraction unit to perform segment-level feature extraction and encoding on the input video;

[0008] S2. Use a foreground mask generation module to model the global temporal correlation of the video and predict and generate a mask of the action foreground (action subject and boundary);

[0009] S3. Use the self-attention mechanism in the mask-guided Transformer module to calculate semantic-level temporal correlation, extract the features of the action while suppressing background interference;

[0010] S4. Use a temporal action localization prediction output module to predict an action integrity map and a boundary confidence map for temporal action proposal and evaluation;

[0011] S5. Train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain a video temporal action localization method based on semantic-level correlation modeling;

[0012] S6. Use the video temporal action localization method based on semantic-level correlation modeling to locate action targets in a new video sequence.

[0013] Preferably, the step S1 specifically includes the following steps:

[0014] S11. Regard the original unclipped long video as a frame set Then divide it into a segment set at regular frame intervals δ where l s = l v / δ;

[0015] S12. Input each segment into a pre-trained two-stream network TSN to extract the features of each segment In this way, a segment feature sequence is obtained

[0016] S13. Use linear interpolation to unify each video to the same temporal length, and the video feature extraction unit outputs video features represented as , which contains T feature units and is shared by subsequent modules.

[0017] Preferably, the step S2 specifically includes the following steps:

[0018] S21. Use the global perception attention module to model the global temporal correlation of the video; input the video feature F, add sine position encoding, and then project the video feature into three different feature representations through a linear transformation, namely query key and value Use these three features for self-attention calculation to learn the temporal interaction between video segments, and output the obtained by global correlation modeling. Input it into a feed-forward neural network with a ReLU activation function, residual connection, and layer normalization, and finally output the features modeled by the global context

[0019] S22. Use the features F g modeled by the global context to predict the foreground mask of the action subject and the foreground mask of the action boundary; first, use stacked one-dimensional temporal convolutions and the Sigmoid activation function to predict the probabilities of the action subject, start, and end of each video segment Secondly, set a threshold for the probability sequence for binarization operation to obtain the foreground mask sequence The i-th element S i ∈ {0, 1} in these sequences represents the foreground binary probability of the i-th segment. Then, copy the foreground mask sequence T times in the temporal dimension to obtain a two-dimensional foreground mask map; finally, in order to introduce the action movement trend information around the foreground, use a dilation kernel to perform a dilation operation on the two-dimensional mask map to obtain the dilated foreground mask map

[0020] Preferably, the step S21 is specifically: add sine position encoding to the input video feature to obtain the video feature F' after position encoding; use this feature for self-attention calculation, and the calculation process is specifically:

[0021] Q = W Q F′, K = W K F′, V = W V F′

[0022]

[0023] where ​are learnable weights in the network, d = C / M represents the dimension of each attention head, and LN represents layer normalization. The self-attention mechanism outputs video features for global correlation modeling Then it is added element-wise to the feature F', and then fed into a feed-forward neural network with a ReLU activation function to enhance the non-linear expression ability of the features. The result is added element-wise to obtain features for global context modeling

[0024] Preferably, in the foreground mask prediction process of step S22, the binarization threshold of the action subject is set to 0.4, the binarization threshold of the boundary mask is set to 0.5, and the size of the dilation kernel is set to kernel=(3,3).

[0025] Preferably, step S3 specifically includes the following steps:

[0026] S31. Add positional encoding to the input video feature sequence F, and perform linear mapping to map it to four new feature matrices respectively representing the query feature, value feature, action start key feature, and action end key feature of the video; then, calculate the cosine similarity between the query feature and the two key features, and output the action start similarity and the action end similarity

[0027] S32. Based on the foreground mask map, construct a masking operation M(·), and then take the Hadamard product of the masked result with the action start similarity SM s and the action end similarity SM e to obtain a mask-guided mapping

[0028] S33. Use the Softmax function to normalize the mask-guided mapping, and then multiply it with the value matrix to capture semantic associations from the action segments of the video to obtain action start and action end enhanced features Concatenate the action start and action end enhanced features and use a one-dimensional convolutional layer to reduce the number of channels of the features from 2C' to C; use residual connection and a feed-forward neural network to enhance non-linearity to obtain boundary enhanced features Use the action subject foreground mask map to obtain action subject enhanced features

[0029] Preferably, the calculation process of the cosine similarity is as follows:

[0030]

[0031] The mask-guided process is as follows: First, construct a masking operation, and then calculate the Hadamard product of the masked result and the action similarity mapping, which is expressed by the formula:

[0032]

[0033] where M(·) represents the masking operation, represents the Hadamard product.

[0034] Preferably, step S4 specifically includes the following steps:

[0035] S41. In the action integrity regression module, input the action subject enhanced feature F a , first use a one-dimensional convolution to predict the action probability of each frame to obtain the action probability feature Secondly, perform feature sampling on the start, center, and end regions of all candidate proposals on P a to construct an action integrity feature map where T×T×N represents all candidate proposals at all arbitrary start and end times, and N is the number of sampling points for each proposal; finally, input FM a into a stacked two-dimensional convolutional neural network to perform a convolution operation, and use a Sigmoid activation function to predict the action integrity confidence map

[0036]

[0037] S42. In the action boundary classification module, input the action boundary enhanced feature F b , first upsample on F bf to construct a boundary feature map of candidate proposals containing the boundary features of T×T candidate proposals; then use a three-dimensional convolutional layer to fuse the features of the sampling points; finally, use a stacked two-dimensional convolutional layer to predict and output the action boundary confidence map of candidate proposals

[0038] Preferably, step S5 specifically includes the following steps:

[0039] S51. Use the server to execute the video feature extraction unit to obtain the feature sequence F of the video;

[0040] S52. Use the server to execute the foreground mask generation module to predict and output the action subject foreground mask map and the action boundary foreground mask map M a , M s , M e ;

[0041] S53: Using the server to execute the mask-guided Transformer module, based on the foreground mask image, using the self-attention mechanism to model the semantic-level temporal association between each frame and the action segment, and output the action subject enhanced feature F. a and action boundary enhancement feature F b ;

[0042] S54, using the server to execute the temporal action positioning prediction output module, and the action subject enhancement feature F a Input to the action integrity regression module to predict the action integrity confidence map P c ; At the same time, the action boundary enhancement feature F b Send it to the boundary classification module and output the action boundary confidence map P s,e ;

[0043] S55. Use the server to perform network training. The mathematical expression of the overall loss function of the training model in an end-to-end manner is:

[0044] L=L mask +λ1L comple +λ2L start +λ3L end

[0045] Among them, L mask =L bl (G a ,S a )+L bl (G s ,S s )+L bl (G e ,S e ) is the loss of the mask generation module, and its weighted balance parameter is set to 1; is the loss of the action integrity module, and its weighted balance parameter is set to λ1 = 1; is the action onset boundary classification loss, is the action end boundary classification loss, and their weighted balance factor is λ2=λ3=1;

[0046] S56. Use the server to optimize the objective function and obtain the local optimal network parameters; use the loss function L in step S55 as the objective function, and use the AdamW optimizer to iteratively update the network parameters to reduce the target loss function until it converges to the local optimum. At this point, the training is completed, and the weights of the trained video temporal action localization network based on semantic-level temporal association modeling are obtained.

[0047] Preferably, step S6 specifically includes the following steps:

[0048] S61. First, send the new video to the video feature extraction unit to extract video features. Then, use the foreground mask generation module to generate the foreground mask map of the actions in the video. After that, input the video features and the foreground mask map into the mask-guided Transformer module to establish the semantic-level temporal correlation of the video and output the video features with enhanced action subjects and action boundaries. Finally, send the video features with enhanced action subjects and action boundaries to the temporal action localization prediction output module to output the action integrity confidence map P c and the action boundary confidence map P s,e ;

[0049] S62. In the upper right corner area of the action integrity confidence map P c and the action boundary confidence map P s,e , calculate the average value of each row of P c and each column of P s,e to obtain the action start score sequence and the action end score sequence . Then, fuse the action start score i , the action end score j and the action integrity score of each candidate action proposal (t , t ). Specifically:

[0050] S63. Adopt the Soft-NMS algorithm to eliminate redundancy through the score decay function to obtain a sparse set of candidate action proposals where P i,j ' is the final confidence score and N' is the number of final candidate proposals;

[0051] S64. Multiply the top two action category labels and the corresponding scores of the new video predicted by UntrimmedNet by the confidence scores of the candidate action proposals output in step S63 to obtain the final temporal action localization result for this video where (t i , t j ) represents the video segment with the predicted action start time of t i and the end time of t j . Its corresponding action category is c and the confidence score is S i,j .

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] A video temporal action localization method based on semantic-level temporal correlation modeling provided by the present invention completes the video temporal action localization task by modeling the semantic-level temporal correlation between each frame in the video and the action segment, and learning to extract more effective video features. The attention mechanism of the Transformer is guided by the foreground mask to calculate the semantic-level similarity between each frame and the foreground, capturing semantic associations. The foreground prediction can refine its action boundary according to the learned action clues; at the same time, the background prediction can suppress false positive predictions and background interference according to the learned semantic gap. The action segments in the video can be accurately located in many difficult actual scenarios, with the AUC value reaching 68.85% on the ActivityNet validation set and the average detection accuracy mAP of all categories reaching 48.4% on the THUMOS validation set, exceeding most existing algorithms and having very good results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is an algorithm framework diagram of a video temporal action localization method based on semantic-level temporal correlation modeling of the present invention;

[0055] Figure 2 It is a flowchart of predicting the foreground mask map from the foreground probability sequence of the present invention;

[0056] Figure 3 It is a unit diagram of the mask-guided Transformer module of the present invention;

[0057] Figure 4 It is a unit diagram of the temporal action localization prediction output module of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] Aiming at the problems and deficiencies in the prior art, the present invention designs a brand-new deep neural network framework for video temporal action localization. It uses the foreground mask as prior knowledge to model the semantic-level association between each frame and the action segment, thereby enhancing the feature representation of the video and further improving the accuracy of action localization. First, we design a foreground mask generation module to adaptively generate a foreground mask, representing the positions of the frames related to the action in the video. This foreground mask is used subsequently to guide the Transformer module to model the semantic-level association with the action frames. Then, we propose a mask-guided Transformer module, which uses the foreground mask to guide the attention mechanism to learn semantic-level similarities, constructs semantic intra-similarities for the foreground to mine supportive clues from other action segments, and at the same time models the semantic gap between the background segment and the action frame to better distinguish the background content. Finally, we integrate the two modules in a unified framework to implement an end-to-end deep neural network for video temporal action localization.

[0060] A video temporal action localization method based on semantic-level temporal association modeling proposed by the present invention includes the following steps:

[0061] S1. Use the video feature extraction unit to perform segment-level feature extraction and encoding on the input video;

[0062] S2. Use the foreground mask generation module to model the global temporal association of the video, and predict and generate a mask of the action foreground (action subject and boundary); representing the positions of the frames related to the action in the entire video, and used subsequently for the mask-guided Transformer module to model the semantic-level association;

[0063] S3. Use the self-attention mechanism in the mask-guided Transformer module to calculate the semantic-level temporal association, that is, calculate the semantic-level similarity between each frame and the action segment, extract the features of the action while suppressing background interference;

[0064] S4. Use the temporal action localization prediction output module to predict the action integrity map and the boundary confidence map for temporal action proposal and evaluation;

[0065] S5. Train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain a video temporal action localization method based on semantic-level association modeling;

[0066] S6. Use the video temporal action localization method based on semantic-level association modeling to locate the action target in a new video sequence.

[0067] The following is a detailed description of each step.

[0068] Step S1: Use the video feature extraction unit to perform segment-level feature extraction and encoding on the input video. As shown in the Ⅰ unit in Figure 1 , it takes the original long video as input, divides the video into small segments, extracts features for each segment, and thus obtains the feature sequence of the video. The main steps are as follows:

[0069] S11: Regard the original unclipped long video as a set of frames Then divide it into a set of segments at regular frame intervals δ where l s = l v / δ;

[0070] S12: Input each segment into the pre-trained two-stream network TSN to extract the features of each segment In this way, a segment feature sequence is obtained

[0071] S13: Use linear interpolation to unify each video to the same temporal length, and the output of the video feature extraction unit is expressed as video features, which contain T feature units and are shared by subsequent modules.

[0072] Step S2: Use the foreground mask generation module to model the global temporal correlation of the video and predict and generate the mask of the action foreground (action subject and boundary); it represents the positions of the frames related to the action in the whole video, and is used later to guide the Transformer module to model the semantic-level correlation. As shown in the Ⅱ module in Figure 1 , the foreground mask generation module unit designed by the present invention is shown, and the main steps are as follows:

[0073] S21: Use the global perception attention module to model the global temporal correlation of the video; input the video feature F, add sinusoidal position encoding, and then project the video feature into three different feature representations by linear transformation, namely query key and value Use these three features for self-attention calculation, learn the temporal interaction between video segments, and model the global temporal correlation. Specifically:

[0074]

[0075] where d = C / M represents the dimension of each attention head, and LN represents layer normalization;

[0076] The video feature output by the self-attention mechanism is input into the feed-forward neural network and passed through a residual connection to obtain the video feature of global context modeling Specifically:

[0077]

[0078] Among them, MLP represents a feed-forward neural network with a ReLu activation function;

[0079] S22. Feature F modeled using global context g Predict the foreground mask of the action subject and the foreground mask of the action boundary; first, use three parallel one-dimensional temporal convolutions and the Sigmoid activation function to predict the probabilities of the action subject, start, and end of each video segment As Figure 2 shown in the flowchart for predicting the foreground mask map from the foreground probability sequence. Secondly, generate the action foreground mask map based on the probability sequence. Specifically, set a threshold for the probability sequence for binarization. The implementation method is to judge whether each small segment is greater than a certain threshold. If it is greater, binarize the probability of this segment to 1, otherwise to 0. Finally, replicate the foreground mask sequence in the temporal dimension to obtain the foreground mask map. These processes are expressed by the formula:

[0080]

[0081] M a ,M s ,M e =Φ(rearrange(S a ,S s ,S e ))

[0082] Among them, H m represents a stacked one-dimensional temporal convolution layer; α m is the binary probability threshold for the foreground. The binarization threshold for the action subject is set to 0.4, and the binarization threshold for the action boundary is set to 0.5. Rearrange represents the replication operation on the mask sequence; Φ is the dilation convolution kernel, and the size of the kernel is set to kernel=(3,3);

[0083] To use the foreground mask to guide the subsequent attention mechanism of the Transformer to model semantic-level temporal associations, since the attention map is two-dimensional and the size of each dimension is the length T of the video, the foreground mask sequence is replicated T times in the temporal dimension to obtain a two-dimensional foreground mask map. This foreground mask map represents the positions of the frames related to the action; finally, to introduce information about the action change trend around the action segment, the mask map is dilated using a 3×3 dilation kernel to obtain the dilated foreground mask map

[0084] Step S3: Use the self-attention mechanism in the mask-guided Transformer module to calculate the semantic-level temporal correlation, that is, calculate the semantic-level similarity between each frame and the action segment, extract the features of the action while suppressing background interference. As Figure 3 shown in the unit diagram of the mask-guided Transformer module designed by the present invention, the main steps are as follows:

[0085] S31: Add positional encoding to the input video feature sequence F, and use learnable linear mapping to map it to four new feature matrices Among them, they respectively represent the query feature, value feature, action start key feature and action end key feature of the video; then, calculate the cosine similarity between the query feature and the two key features, and output the action start similarity and the action end similarity

[0086] S32: According to the foreground mask map generated in step S22, construct the mask operation M(·), specifically:

[0087]

[0088] Then calculate the Hadamard product of the mask result and the action start similarity SM s and the action end similarity SM e to obtain the mask-guided mapping This process realizes using the foreground mask to guide the attention mechanism of the Transformer to model the semantic-level connection;

[0089] S33: Use the Softmax function to normalize the mask-guided mapping, and then multiply it with the value feature matrix of the video to obtain the action start and action end enhanced features captures the semantic association of the action segment of the video; in this process, for the foreground segment, it can learn the relevant information and support clues of other action segments in the video, and for the background segment, it can capture the semantic gap between it and the action frame; then, splice these two types of features and send them into a one-dimensional convolutional layer for information fusion, and reduce the number of channels from 2C' to C; then add this feature to the video feature with positional encoding element by element, and send it into a feed-forward neural network to enhance the non-linear expression ability of the feature, and output the boundary enhanced feature Similarly, the action mask-guided Transformer module outputs the action subject enhanced feature

[0090] Among them, the calculation process of the cosine similarity in step S31 is as follows:

[0091]

[0092] The mask-guided process is as follows: First, construct a masking operation, and then calculate the Hadamard product of the mask result and the action similarity mapping. The formula is expressed as:

[0093]

[0094] where M(·) represents the masking operation, represents the Hadamard product.

[0095] Specifically, step S33 is specifically: Normalize the mask-guided mapping, multiply it with the value matrix to capture the semantic association from the action segments of the video, and output the action start and action end enhanced features <![CDATA[ ]] Concatenate the two and send them into a one-dimensional convolution to compress the channel dimension, and then add them element-wise to the original video features; send the result into a feed-forward neural network, and output the boundary enhanced features after passing through the residual link. The formulaic representation of these processes is:

[0096]

[0097] where [·] represents the concatenation operation, FFN represents the feed-forward neural network, represents element-wise addition.

[0098] Step S4: Use the temporal action localization prediction output module to predict the action integrity map and the boundary confidence map for temporal action proposal and evaluation. As Figure 4 shown in the unit diagram of the temporal action localization prediction output module designed by the present invention, the main steps are:

[0099] S41: In the action integrity regression module, input the action subject enhanced feature F a , first use a one-dimensional convolution to predict the action probability for each frame to obtain the action probability feature Secondly, perform feature sampling on all candidate proposals within the start, center, and end regions of P a to construct the action integrity feature map where T×T×N represents all candidate proposals for all arbitrary start and end times, and N is the number of sampling points for each proposal; set N = 32. Finally, input FM a into the stacked two-dimensional convolutional neural network to perform convolution operations, and use the Sigmoid activation function to predict the action integrity confidence map It represents the action integrity scores of all candidate proposals; this process is specifically expressed as:

[0100] P a = F (Conv1D) (Faf )

[0101] FM a = Sample(P a )

[0102] P c = Sigmoid(F (Conv2D) (FM a ))

[0103] where F (Conv1D) (·) is a one-dimensional convolution operation, F (Conv2D) (·) is a two-dimensional convolution operation, and Sample(·) represents the feature sampling process;

[0104] S42. In the action boundary classification module, input the action boundary enhanced feature F b , first upsample F bf to construct the boundary feature map of candidate proposals containing the boundary features of T×T candidate proposals. The number of sampling points is also N = 32. Then use a three-dimensional convolution layer to fuse and compress the features of 32 sampling points into N = 1; finally, use a stacked two-dimensional convolution layer to predict and output the action boundary confidence map of candidate proposals This process is specifically expressed as:

[0105] FM b = Sample(F bf )

[0106] P s,e = Sigmoid(F Conv2D (F Conv3D (FM b )))

[0107] where F (Conv3D) (·) represents a three-dimensional convolution operation.

[0108] Step S5. Train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain a video temporal action localization method based on semantic-level association modeling. As Figure 1 shown is the algorithm framework diagram of a video temporal action localization method based on semantic-level temporal association modeling of the present invention. The main steps are:

[0109] S51. Input the videos in the training set into the server, execute the video feature extraction unit, and obtain the feature sequence F of the videos;

[0110] S52. Use the server to execute the foreground mask generation module to predict and output the action subject foreground mask map and the action boundary foreground mask map M a , M s , Me ;

[0111] S53. Use the server to execute the masked-guided Transformer module. Based on the foreground mask map, use the self-attention mechanism to model the semantic-level temporal association between each frame and the action segment, and output the action subject enhanced feature F a and the action boundary enhanced feature F b ;

[0112] S54. Use the server to execute the temporal action localization prediction output module. Input the action subject enhanced feature F a to the action integrity regression module to predict the action integrity confidence map P c ; At the same time, send the action boundary enhanced feature F b to the boundary classification module to output the action boundary confidence map P s,e ;

[0113] S55. Use the server for network training, and train in an end-to-end manner; The mathematical expression of the loss of the foreground mask generation module is:

[0114] L mask = L bl (G a , S a ) + L bl (G s , S s ) + L bl (G e , S e )

[0115] where is the weighted binary logistic regression loss; G a , G s , G e represent the true action foreground mask, α + = T / ∑(g i ) and α - = T / ∑(1 - g i ) are the weight balance factors;

[0116] The mathematical expression of the action integrity regression loss is:

[0117] <tmp <tmp

[0118] where represents the maximum value of the IoU between the true proposal and the action instance, and P a is the true action probability sequence;

[0119] The mathematical expression of the action integrity regression loss is:

[0120]

[0121] Among them, represents the confidence of the true action boundary probability;

[0122] The mathematical expression of the loss function of the overall model is:

[0123] L = L mask + λ1L comple + λ2L start + λ3L end

[0124] where λ1, λ2, and λ3 are hyperparameters used to balance the three losses, and are set to λ1, λ2, λ3 = 1;

[0125] S56. Optimize the objective function using the server to obtain local optimal network parameters; use the loss function L in step S55 as the objective function, and use the AdamW optimizer to iteratively update the network parameters to reduce the objective loss function until it converges to the local optimum. At this point, the training ends, and the weights of the trained video temporal action localization network based on semantic-level temporal correlation modeling are obtained.

[0126] Step S6. Locate the action target in a new video sequence using the video temporal action localization method based on semantic-level correlation modeling. The main steps are as follows:

[0127] S61. First, send the new video into the video feature extraction unit to extract video features; then use the foreground mask generation module to generate the foreground mask map of the action in the video; then input the video features and the foreground mask map into the mask-guided Transformer module to establish the semantic-level temporal correlation of the video, and output the video features with enhanced action subject and action boundary; finally, send the video features with enhanced action subject and action boundary into the temporal action localization prediction output module to output the action integrity confidence map P c and the action boundary confidence map P s,e ;

[0128] S62. In the upper right corner area of the action integrity confidence map P c and the action boundary confidence map P s,e , calculate the average value of each row of P c and each column of P s,e to obtain the action start score sequence and the action end score sequence Then, for each candidate action proposal (t i , t j ), the action start score action end score action integrity score Fusion is performed as follows:

[0129] S63. The Soft-NMS algorithm is adopted to eliminate redundancy through a score decay function, and a sparse set of candidate action proposals is obtained where P i,j ' is the final confidence score, and N' is the number of final candidate proposals;

[0130] S64. The top two action category labels and corresponding scores of the new video predicted by UntrimmedNet are multiplied by the confidence scores of the candidate action proposals output in step S63 to obtain the final temporal action localization result for the video where (t i , t j ) represents a video segment with a predicted action start time of t i and an end time of t j . Its corresponding action category is c, and the confidence score is S i,j .

[0131] Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed, as long as they do not depart from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the different dependent claims and the features described herein can be combined in a manner different from that described in the original claims. It should also be understood that the features described in connection with a single embodiment can be used in other described embodiments.

Claims

1. A video temporal action localization method based on semantic-level temporal correlation modeling, characterized in that It includes the following steps: S1. Use the video feature extraction unit to perform segment-level feature extraction and encoding on the input video; S2. Use the foreground mask generation module to model the global temporal correlation of the video, and predict and generate the mask of the action foreground, where the action foreground is the action subject and boundary; S3. Use the self-attention mechanism in the mask-guided Transformer module to calculate the semantic-level temporal correlation, extract the features of the action while suppressing background interference; S4. Use the temporal action localization prediction output module to predict the action integrity map and the boundary confidence map for temporal action proposal and evaluation; S5. Train the network model, optimize the network parameters by reducing the network loss function until the network converges, and obtain the video temporal action localization method based on semantic-level correlation modeling; S6. Use the video temporal action localization method based on semantic-level correlation modeling to locate the action target in a new video sequence; The specific steps of step S3 include the following steps: S31. Add positional encoding to the input video feature sequence F, and perform linear mapping to map it to four new feature matrices which respectively represent the query feature, value feature, action start key feature, and action end key feature of the video; then, calculate the cosine similarity between the query feature and the two key features, and output the action start similarity and the action end similarity S32. Based on the foreground mask map, construct a masking operation M(·), and then take the Hadamard product of the masking result with the action start similarity SM s , the action end similarity SM e to obtain a mask-guided mapping GM s , S33. Use the Softmax function to normalize the mask-guided mapping, and then multiply it with the value matrix to capture semantic associations from the action segments of the video to obtain the action start and action end enhanced features F s , Concatenate the action start and action end enhanced features and use a one-dimensional convolutional layer to reduce the number of channels of the features from 2C' to C; use residual connections and a feedforward neural network to enhance the non-linearity to obtain the boundary enhanced features Use the action subject foreground mask map to obtain the action subject enhanced features 2. The video temporal action localization method based on semantic-level temporal correlation modeling according to claim 1, wherein The specific steps of step S1 include the following steps: S11. Regard the original unclipped long video as a set of frames Then divide it into a set of segments at regular frame intervals δ where l s = l v / δ; S12. Input each segment into the pre-trained two-stream network TSN to extract the features of each segment In this way, a sequence of segment features is obtained S13. Use linear interpolation to unify each video to the same temporal length, and the video feature extraction unit outputs video features represented as , which contains T feature units and is shared by subsequent modules.

3. A video temporal action localization method based on semantic-level temporal correlation modeling according to claim 1, characterized in that The specific steps of step S2 include the following steps: S21. Model the global temporal correlation of the video using the global perception attention module; input the video feature F, add sine position encoding, and then project the video feature into three different feature representations, namely query key and value Use these three features for self-attention calculation to learn the temporal interaction between video segments, and output the Project the into a feed-forward neural network with ReLU activation function, residual connection, and layer normalization, and finally output the feature modeled by the global context S22. Feature F modeled using global context g Predict the foreground mask of the action subject and the foreground mask of the action boundary; first, use stacked one-dimensional temporal convolutions and the Sigmoid activation function to predict the probabilities of the action subject, start, and end of each video segment Secondly, set a threshold for the probability sequence for binarization operation to obtain the foreground mask sequence S a , S s , Then, copy the foreground mask sequence T times in the temporal dimension to obtain a two-dimensional foreground mask map; finally, in order to introduce the action motion trend information around the foreground, use a dilation kernel to perform a dilation operation on the two-dimensional mask map to obtain the dilated foreground mask map M a , M s , 4. A video temporal action localization method based on semantic-level temporal correlation modeling according to claim 3, characterized in that The specific step S21 is: Add sine position encoding to the input video features to obtain the video features F' after position encoding; use these features for self-attention calculation, and the calculation process is specifically: Q = W Q F′, K = W K F′, V = W V F′ Among them, W Q , W K , are learnable weights in the network, d = C / M represents the dimension of each attention head, and LN represents layer normalization. The self-attention mechanism outputs video features for global correlation modeling Then it is added element-wise with the feature F', and then fed into a feed-forward neural network with a ReLU activation function to enhance the non-linear expression ability of the features. The result is added element-wise to obtain features for global context modeling 5. The video temporal action localization method based on semantic-level temporal correlation modeling according to claim 3, characterized in that In the foreground mask prediction process of step S22, the binary threshold of the action subject is set to 0.4, the binary threshold of the boundary mask is set to 0.5, and the size of the dilation kernel is set to kernel=(3,3).

6. The video temporal action localization method based on semantic-level temporal correlation modeling according to claim 5, characterized in that The calculation process of the cosine similarity is: The mask-guided process is: First construct the mask operation, and then calculate the Hadamard product of the mask result and the action similarity mapping, and the formula is expressed as: Where M(·) represents the mask operation, and ° represents the Hadamard product.

7. A video temporal action localization method based on semantic-level temporal correlation modeling according to claim 5, characterized in that The specific steps of step S4 include the following steps: S41. In the action integrity regression module, input the enhanced feature F of the action subject a , first use one-dimensional convolution to predict the action probability of each frame to obtain the action probability feature Secondly, perform feature sampling on the start, center, and end regions of all candidate proposals on P a to construct the action integrity feature map where T×T×N represents all candidate proposals at all arbitrary start and end times, and N is the number of sampling points for each proposal; finally, input FM a into the stacked two-dimensional convolutional neural network to perform convolution operations, and use the Sigmoid activation function to predict the action integrity confidence map S42. In the action boundary classification module, input the action boundary enhanced feature F b , first, upsample F bf to construct a boundary feature map of candidate proposals that contains boundary features of T×T candidate proposals; then use a three-dimensional convolutional layer to fuse the features of the sampling points; finally, use stacked two-dimensional convolutional layers to predict and output an action boundary confidence map of the candidate proposals 8. A video temporal action localization method based on semantic-level temporal correlation modeling according to claim 7, characterized in that The specific steps of step S5 include the following steps: S51. Use the server to execute the video feature extraction unit to obtain the feature sequence F of the video; S52. Use the server to execute the foreground mask generation module to predict and output the foreground mask map of the action subject and the foreground mask map of the action boundary M a ,M s ,M e ; S53. Use the server to execute the masked guided Transformer module. Based on the foreground mask graph, use the self-attention mechanism to model the semantic-level temporal association between each frame and the action segment, and output the enhanced feature F of the action subject a and the enhanced feature F of the action boundary b ; S54. Use the server to execute the timing action localization prediction output module, and input the action subject enhancement feature F a to the action integrity regression module to predict the action integrity confidence map P c ; at the same time, input the action boundary enhancement feature F b into the boundary classification module to output the action boundary confidence map P s,e ; S55. Use the server to perform network training. The mathematical expression of the overall loss function of the model trained in an end-to-end manner is: L = L mask + λ1L comple + λ2L start + λ3L end Among them, L mask = L bl (G a , S a ) + L bl (G s , S s ) + L bl (G e , S e ) is the loss of the mask generation module, and its weighted balance parameter is set to 1; is the loss of the action integrity module, and its weighted balance parameter is set to λ1 = 1; is the action start boundary classification loss, is the action end boundary classification loss, and their weighted balance factors are λ2 = λ3 = 1; S56. Use the server to optimize the objective function to obtain the local optimal network parameters; use the loss function L in step S55 as the objective function, and use the AdamW optimizer to iteratively update the network parameters to reduce the objective loss function until it converges to the local optimum. At this point, the training ends, and the weights of the trained video temporal action localization network based on semantic-level temporal correlation modeling are obtained.

9. A video temporal action localization method based on semantic-level temporal correlation modeling according to claim 8, characterized in that The specific steps of step S6 include the following steps: S61. First, send the new video to the video feature extraction unit to extract video features; then use the foreground mask generation module to generate a foreground mask map of the actions in the video; after that, input the video features and the foreground mask map into the mask-guided Transformer module to establish the semantic-level temporal correlation of the video, and output the video features with enhanced action subjects and action boundaries; finally, send the video features with enhanced action subjects and action boundaries to the temporal action localization prediction output module to output the action integrity confidence map P c and the action boundary confidence map P s,e ; S62. In the upper right corner region of the action integrity confidence map P c and the action boundary confidence map P s,e , for each row of P c and each column of P s,e , calculate the average value to obtain the action start score sequence and the action end score sequence Then, fuse the action start score i , the action end score j and the action integrity score of each candidate action proposal (t , t ), specifically as follows: S63. The Soft-NMS algorithm is adopted to eliminate redundancy through a score decay function, obtaining a sparse set of candidate action proposals. where P i,j ' is the final confidence score and N' is the number of final candidate proposals; The top two action category labels and corresponding scores of the new video predicted by UntrimmedNet are multiplied by the confidence score of the candidate action proposal output in step S63 to obtain the final temporal action localization result for this video where (t i , t j ) represents a video segment with a predicted action start time of t i and an end time of t j . Its corresponding action category is c, and the confidence score is S i,j .

Images