A method for locating a time sequence action based on a typical action network

Abstract

The application discloses a kind of time sequence action positioning methods based on typical action network, including step 1: extracting video features from given video;Step 2: according to video features, typical action is generated using clustering algorithm;Step 3: according to the information transmission relationship between typical action, construct typical action network module based on structure information transmission;Step 4: typical action network module is combined with time sequence action positioning method ActionFormer;Step 5: classification estimation based on similarity comparison;Step 6: training and testing;The method can effectively improve the representation and discriminability of the algorithm by combining the typical action network module with the existing time sequence action positioning method, has the characteristics of good representation and discriminability, accurate positioning.

Description

Technical Field

[0001] This invention relates to the field of computer vision algorithm research technology, specifically to a temporal action localization method based on a typical action network. Background Technology

[0002] In the information age of the Internet of Everything, video, as an effective medium for recording the objective world and transmitting information, is widely used in production and daily life, which makes the demand for video interpretation increasingly strong.

[0003] To effectively perceive the structure of actions, researchers in the field of video action interpretation have conducted in-depth explorations and achieved a series of results. In their 2020 work, *G-tad: Sub-graph localization for temporal action detection*, Mengmeng Xu et al. modeled the temporal and semantic neighborhood relationships of video segments. In their 2021 work, *Graph convolutional module for temporal action localization in videos*, Runhao Zeng et al. represented the relationships between neighboring action proposals using graph topology. Recently, self-attention mechanisms have demonstrated excellent action structure modeling capabilities. Among them, Ashish Vaswani et al.'s 2017 work, *Attention is all you*, further illustrates this. In Need, dense connections enable video frames to interact with any neighboring frames after only one serialization operation, and the sufficient interaction of neighboring frames can provide rich temporal context information for the video frame. However, graph convolution and self-attention mechanisms aim to learn a unique and definite representation for each type of action, but they usually ignore intra-class diversity and ignore that an action usually contains multiple stages, and the video frames of each stage also exhibit different appearances and motion characteristics. In addition, existing methods are usually limited to a single video, or even a sliding window in the video, to model the action structure, and have not yet fully explored cross-video category-level relationships.

[0004] To characterize the diversity of similar actions and model action category-level relationships across videos, researchers have explored several meaningful methods. For example, Wang Luo et al. proposed a learning memory bank in their 2021 work, "Action unit memory network for weakly supervised temporal action localization," which serves as a prototype for action learning categories. Le Yang et al., in their 2022 work, "Colar: Effective and efficient online action detection by consulting exemplars," selected representative sub-actions from each action category and treated them as typical action segments. These models have also achieved corresponding performance improvements on specific tasks. However, the aforementioned methods have poor representational and discriminative power and cannot effectively serve video interpretation tasks involving action structures.

[0005] Therefore, there is an urgent need to design a temporal action localization method based on typical action networks to improve the representation and discrimination capabilities in video interpretation tasks, thereby solving the problems existing in the above-mentioned technologies. Summary of the Invention

[0006] To address the aforementioned problems, this invention aims to provide a temporal action localization method based on a typical action network. This method considers an action category. First, it collects each feature of all action instances within that category and performs clustering, selecting representative features as typical action segments. Then, it divides these typical action segments into three stages and embeds them into a directed graph in chronological order, globally constructing a typical action network at the category level. Next, it designs four information interaction steps within the typical action network: intra-stage interaction, adjacent stage transmission, typical information aggregation, and video frame broadcasting. These four steps process temporal information sequentially, adjusting the features of each typical action segment in the typical action network to provide appropriate temporal information guidance for each processed video frame. Finally, the typical action network module is combined with existing temporal action localization methods to achieve performance gains, effectively improving the module's representational and discriminative power, resulting in good representational and discriminative power and accurate localization.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A temporal action localization method based on a typical action network includes...

[0009] Step 1: Extract video features from the given video;

[0010] Step 2: Generate typical actions using a clustering algorithm based on video features;

[0011] Step 3: Based on the information transmission relationships between typical actions, construct typical action network modules based on structural information transmission;

[0012] Step 4: Combine the typical action network module with the actionformer temporal action localization method;

[0013] Step 5: Perform classification estimation based on similarity comparison on the results of combining typical action network modules with the temporal action localization method ActionFormer;

[0014] Step 6: Set the loss function and train and test the classification results.

[0015] Preferably, the process of extracting video features from a given video in step 1 is as follows:

[0016] Given a video, the I3D model is used to extract a series of video features from the video to obtain a high-level semantic representation of the video:

[0017] F = [f1, f2, ..., f T ],

[0018] in, Let f represent the t-th video feature. t It reflects information from a continuous video frame. Preferably, step 2, which involves generating typical actions using a clustering algorithm, includes...

[0019] Step 2.1: Consider a class of actions, collect the feature representations of all action instances, and use the K-Means algorithm to cluster the features to obtain M clusters; for each cluster feature representation, take the feature closest to the cluster center as the typical feature to obtain M typical features;

[0020] Step 2.2: After obtaining M typical features, divide each action instance evenly into three stages: action beginning, action evolution, and action ending.

[0021] Preferably, the process of constructing a typical action network module in step 3 includes:

[0022] Step 3.1: Connect multiple typical actions belonging to the same stage, and perform bidirectional information exchange between every two typical actions;

[0023] Step 3.2: Information transfer from "action start" to "action evolution" and from "action evolution" to "action end"; connect each typical feature of the start stage with all typical features of the evolution stage, connect each typical feature of the evolution stage with all typical features of the end stage, and finally, establish a directed graph between typical features of different stages.

[0024] Step 3.3: Aggregate typical information into video frames to provide temporal information support for the action interpretation task; at the same time, compare the video frame features with all typical features one by one, and use an attention mechanism to adaptively aggregate features from each action stage.

[0025] Step 3.4: Use video frame broadcasting to pass the characteristics of the current video frame to the typical action network module; each typical action takes into account its own characteristics and the characteristics of the video frame being processed, and updates its own characteristics according to the attention mechanism.

[0026] Preferably, in the typical action network module described in step 3, the typical actions are executed in the form of a graph transformation network, specifically as follows:

[0027] 1. First, initialize the node and edge features of the graph transformation network.

[0028] (1) Use a linear transformation layer to transform typical action features Projected into the hidden space In addition, consider a path consisting of node v i Pointing to node v j The edge is represented by the characteristic e. ij ;

[0029] (2) During the initialization phase, splice x i and x j The features of the two nodes are analyzed, and a linear transformation layer is applied to project these features. Furthermore, a random walk strategy is employed to generate a positional code for each node, and a linear transformation layer is used to project the positional code into the latent space.

[0030] (3) A typical action network module uses a graph transformation network at each layer. For the graph transformation network of the l-th layer, the feature of the i-th node is: The corresponding position code is Connect node v i and v j The features corresponding to the edges are Using projection matrix To process node features and their corresponding location encodings, a projection matrix is ​​used. To process edge features;

[0031] (4) For the computation node v i and its neighboring node v j , j∈N i Correlation, estimate the correlation vector

[0032]

[0033] in, [·] indicates element-wise multiplication, and [·] indicates concatenation along the channel dimension;

[0034] 2. Calculate similarity using the following formula.

[0035]

[0036] Update node v according to the following formula i Features:

[0037]

[0038] 3. Use standard operations in graph transformation networks to process node features, followed by residual connections, layer-by-layer regularization, and feedforward networks in sequence;

[0039] 4. Taking into account the node's own position code, the position codes of neighboring nodes, and the characteristics of the edges connecting neighboring nodes, update the position code p of each node. i The update process is as follows

[0040]

[0041] Here, Γ(·) represents that in the information propagation process similar to node features, only the positional encoding of the node is considered during the calculation, without considering the feature information of the node.

[0042] Preferably, step 4, which involves combining a typical action network module with the ActionFormer temporal action localization method, includes...

[0043] Step 4.1: Introduce the typical action network module into ActionFormer, and build an additional branch in the network structure of the ActionFormer method;

[0044] Step 4.2: Use max pooling to downsample the features to obtain the same number of representative features as ActionFormer;

[0045] Step 4.3: Using enhanced video features, referencing ActionFormer, solve the temporal action localization task in a one-stage, anchor-free manner.

[0046] Preferably, the classification estimation process based on similarity comparison described in step 5 includes:

[0047] Step 5.1: In a typical action network module, the node features of the last layer graph transform network are represented as follows:

[0048]

[0049] in, The characteristics representing the M typical nodes of class c, The video frame features represented by class c are collected from typical action network modules of all classes to obtain the feature representation.

[0050] Step 5.2: For the action category classification task, collect video frame features from typical action network modules of all categories to obtain feature representations. To obtain the predicted classification score, L2 regularization is applied to all features. Then, the dot product similarity between the video frame features and the canonical features is calculated. The average of the similarity scores for class c is taken to obtain the classification score s. c Finally, the classification score s of the video frame is obtained; for the action boundary regression task, a 3-layer 1D convolutional neural network is used to predict the distance from the current time step to the boundary of the action. and

[0051] in, This represents the distance from the current moment to the start of the action. It represents the distance from the current moment to the moment the action ends.

[0052] Preferably, the training and testing process described in step 6 includes:

[0053] Step 6.1: During the training phase, let the loss function be as follows:

[0054] L = L van +L exp +λ·L cons

[0055] Among them, L van This represents the loss term calculated by the ActionFormer algorithm, and the specific calculation method is as follows:

[0056]

[0057] This represents Focal loss, used for classifying action categories; This represents DIoU loss, used for regression of action boundaries; T + This indicates the number of positive samples in a video. It is an indicator function used to indicate whether the current moment belongs to a certain action; λ reg Represents the equilibrium coefficient, used for balancing. and

[0058] Given the prediction results of typical action branches, calculate the corresponding loss term L for each typical branch using the ActionFormer algorithm. exp The ActionFormer algorithm's predictions are kept consistent with those of typical action branches; the consistency loss term L is calculated. cons λ is used to adjust for the impact of consistency loss;

[0059] Step 6.2: During the testing phase, the prediction results s of the ActionFormer algorithm are fused using the hyperparameter β. van Prediction results of typical action branches exp The final result of the relevant video interpretation task is as follows:

[0060] s = β·s van +(1-β)·s exp .

[0061] The beneficial effects of this invention are: This invention discloses a temporal action localization method based on a typical action network. Compared with the prior art, the improvement of this invention lies in:

[0062] This invention proposes a temporal action localization method based on a typical action network. This method can effectively serve video interpretation tasks involving action structure. As a representative module, firstly, each selected typical action segment corresponds to a series of video frames with clear appearance features and motion patterns. The clear features corresponding to these typical action segments can represent many similar features. Secondly, the typical action network divides a class of actions into three stages: action beginning, action evolution, and action ending, which can explicitly represent the structure of similar actions. Furthermore, each stage of the typical action network contains multiple typical action segments, which can appropriately reflect the appearance and motion diversity of actions in the same stage. As a discriminative module, firstly, compared to the features of most training videos, the typical action network participates in the training of all video frames and is fully trained through interaction with all samples, becoming a discriminative module. In addition, given video features, traditional methods usually learn classifiers with additional parameters to predict classification scores, while the typical action network predicts classification scores by comparing the similarity between video features and each typical action. This approach, which does not introduce additional parameters, can further enhance the feature discriminative power of the typical action network.

[0063] As a module with both representational and discriminative power, the typical action network proposed in this method can be efficiently combined with the ActionFormer method to effectively model action structure and cross-video dependencies, serving the task of temporal action localization. Compared with existing temporal action localization methods, the method of this invention can achieve higher localization accuracy with limited computational overhead. It has the advantages of good representational and discriminative power and accurate localization. Attached Figure Description

[0064] Figure 1 This is a flowchart of the temporal action localization method based on a typical action network according to the present invention.

[0065] Figure 2 This is a visualization of some of the training data in Embodiment 2 of the present invention.

[0066] Figure 3 This is an experimental result diagram of the temporal action localization method based on a typical action network according to Embodiment 2 of the present invention. Detailed Implementation

[0067] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0068] Example 1: As Figures 1-3 This invention presents a temporal action localization method based on typical action networks. The basic idea is as follows: Considering an action category, firstly, this method collects each feature of all action instances of this category and performs clustering, selecting representative features as typical action segments; then, this method divides the typical action segments into three stages and embeds them into a directed graph in chronological order, globally constructing a typical action network at the category level; next, this method designs four information interaction steps in the typical action network: intra-stage interaction, adjacent stage transmission, typical information aggregation, and video frame broadcasting. These four steps process temporal information sequentially, adjusting the features of each typical action segment in the typical action network to provide appropriate temporal information guidance for each processed video frame; finally, the typical action network module is combined with existing temporal action localization methods to achieve performance gains. The specific process includes...

[0069] Step 1: Extract video features

[0070] Given a video, the I3D model is used to extract a series of video features from the video to obtain a high-level semantic representation of the video:

[0071] F = [f1, f2, ..., f T ],

[0072] in, Let f represent the t-th video feature. t It reflects information from a continuous video frame; the I3D model was proposed by Joao Carreira et al. in their 2017 work, Quo vadis, action recognition: a new model and the kinetics dataset [C] / / proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:6299-6308.

[0073] Step 2: Generate typical actions using clustering algorithms based on video features.

[0074] Step 2.1: Consider a class of actions. First, collect the feature representations of all action instances. Then, use the K-Means algorithm to cluster the features to obtain M clusters. For each cluster feature representation, take the feature closest to the cluster center as the typical feature to obtain M typical features.

[0075] The K-Means algorithm was proposed by MacQueen J in his 1967 work, Classification and Analysis of Multivariate Observations;

[0076] Step 2.2: After obtaining M typical features, divide each action instance evenly into three stages: action beginning, action evolution, and action ending; determine the stage to which the typical feature belongs based on its position in the corresponding action instance; for different action categories, the three stages usually contain different numbers of typical actions.

[0077] Step 3: Construct a typical action network module based on structured information transmission

[0078] Given the typical actions corresponding to the three stages, construct a directed graph G using four information transmission mechanisms: intra-stage interaction, adjacent stage transmission, typical information aggregation, and video frame broadcasting. c (V c E c To explicitly represent the structure between typical actions, specifically:

[0079] Step 3.1: First, connect multiple typical actions belonging to the same stage and perform bidirectional information exchange between every two typical actions; considering a specific action stage, multiple typical actions can reflect the diversity of actions in appearance and motion patterns in this stage; therefore, intra-stage interaction facilitates typical actions to capture the diversity in the same temporal segment.

[0080] Step 3.2: The second step includes two types of information transmission: from the "beginning of the action" to the "evolution of the action" and from the "evolution of the action" to the "end of the action".

[0081] Each typical feature in the initial stage is connected to all typical features in the evolution stage, and each typical feature in the evolution stage is connected to all typical features in the final stage. Finally, a directed graph is built between typical features in different stages. The transmission between adjacent stages can help each typical feature to explicitly perceive its predecessor and successor typical features, thereby explicitly representing the structural information of the action.

[0082] Step 3.3: After sufficient information interaction within and between stages, the typical action network can accurately convey structural information to the processed video frames; at this point, the typical information is aggregated into the video frames to provide temporal information support for the action interpretation task; the video frame features are compared with all typical features one by one, and the attention mechanism is used to adaptively aggregate features from each action stage; after the aggregation of typical information, the video frame features can fully perceive the structural information of the current action.

[0083] Step 3.4: Use video frame broadcasting to pass the characteristics of the current video frame to the typical action network module; each typical action takes into account its own characteristics and the characteristics of the video frame being processed, and updates its own characteristics according to the attention mechanism; video frame broadcasting enables typical actions to make adaptive adjustments according to the characteristics of the video frame, thereby generating more representative typical features in the next layer of information propagation.

[0084] Step 4: Combine the typical action network module with the temporal action localization method ActionFormer (proposed by Chen-Lin Zhang et al. in their 2022 paper Zhang CL, Wu J, Li Y. Actionformer: Localizing moments of actions with transformers [C] / / European Conference on Computer Vision. Springer, Cham, 2022:492-510.). The specific steps are as follows:

[0085] Step 4.1: First, introduce the typical action network module into ActionFormer, that is, build an extra branch in the network structure of the ActionFormer method to serve the existing methods in a plug-and-play manner. In the process of interacting with typical actions, the ActionFormer method can provide complete typical action information for each frame to help with action modeling.

[0086] Step 4.2: Then, use max pooling to downsample the features to obtain the same number of representative features as ActionFormer;

[0087] Step 4.3: Next, using enhanced video features, referencing ActionFormer, solve the temporal action localization task in a one-stage, anchor-free manner;

[0088] Step 5: Classification estimation based on similarity comparison

[0089] Step 5.1: In a typical action network module, the node features of the last layer of the graph transform network can be represented as follows:

[0090]

[0091] in, The characteristics representing the M typical nodes of class c, The video frame features represented by class c are used to obtain the feature representation by collecting video frame features from typical action network modules of all classes.

[0092] Step 5.2: For the action category classification task, this method collects video frame features from typical action network modules of all categories to obtain feature representations. To obtain the predicted classification score, L2 regularization is applied to all features. Then, the dot product similarity between the video frame features and the canonical features is calculated. The average of the similarity scores for class c is taken to obtain the classification score s. c Finally, the classification score s of the video frame is obtained;

[0093] For action boundary regression tasks, this method uses a 3-layer 1D convolutional neural network to predict the distance from the current time step to the boundary of the action. and This represents the distance from the current moment to the start of the action. This represents the distance from the current moment to the moment the action ends;

[0094] The kernel size is set to 3, the first two layers use layer normalization, and all three layers use the ReLU activation function.

[0095] Step 6: Training and Testing

[0096] Step 6.1: During the training phase, let the loss function be calculated in the following form:

[0097] L = L van +L exp +λ·L cons

[0098] Among them, L van This represents the loss term calculated by the ActionFormer algorithm, and the specific calculation method is as follows:

[0099]

[0100] This refers to the Focal loss proposed by Tsung-Yi Lin et al. in their 2017 work, "Focal loss for dense object detection," which is used for action category classification. This refers to the DIoU loss proposed by Zhaohui Zheng et al. in their 2020 work, "Distance-IoU loss: Faster and better learning for boundingbox regression," used for regression of action boundaries; T + This indicates the number of positive samples in a video. It is an indicator function used to indicate whether the current moment belongs to a certain action; λ reg Represents the equilibrium coefficient, used for balancing. and

[0101] Given the prediction results of typical action branches, the corresponding loss term L for each typical branch can be calculated using the ActionFormer algorithm. exp The ActionFormer algorithm's predictions are kept consistent with those of typical action branches; the consistency loss term L is calculated. cons λ is used to adjust for the impact of consistency loss;

[0102] Step 6.2: During the testing phase, the prediction results s of the ActionFormer algorithm are fused using the hyperparameter β. van Prediction results of typical action branches exp The final result of the relevant video interpretation task is as follows:

[0103] s = β·s van +(1-β)·s exp .

[0104] Preferably, the typical action network module is implemented based on Deep Graph Library 0.8.2 (Deep Graph Library 0.8.2 was proposed by Minjie Wang et al. in their 2019 work Wang M, Zheng D, Ye Z, et al. Deep graph library: A graph-centric, highly-performant package for graphneural networks[J]. arXiv preprint arXiv:1909.01315,2019.). In the typical action network module, the typical action runs in the form of a graph transformer, specifically as follows:

[0105] 1. First, initialize the node and edge features of the graph transformation network.

[0106] (1) Use a linear transformation layer to transform typical action features Projected into the hidden space In addition, consider a path consisting of node v i Pointing to node v j The edge is represented by the characteristic e. ij ;

[0107] (2) During the initialization phase, splice x i and x j The features of the two nodes are analyzed, and a linear transformation layer is applied to project these features. Furthermore, a random walk strategy is employed to generate a positional code for each node, and a linear transformation layer is used to project the positional code into the latent space.

[0108] The random walk strategy was proposed by Pan Li et al. in their 2020 work, LiP, WangY, WangH, et al. Distance encoding: Design provably more powerful neural networks for graph representation learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 4465-4478.

[0109] (3) A typical action network module uses a graph transformation network at each layer. For the graph transformation network of the l-th layer, the feature of the i-th node is: The corresponding position code is Connect node v i and v j The features corresponding to the edges are Using projection matrix To process node features and their corresponding location encodings, a projection matrix is ​​used. To process edge features;

[0110] (4) For the computation node v i and its neighboring node v j , j∈N i Correlation, estimate the correlation vector

[0111]

[0112] in, [·] indicates element-wise multiplication, and [·] indicates concatenation along the channel dimension;

[0113] 2. Then, calculate the similarity in the following form.

[0114]

[0115] Update node v according to the following formula i Features:

[0116]

[0117] 3. Finally, standard operations in graph transformation networks are used to process node features, successively using residual connections, layer normalization, and feed forward networks. To simplify the description, the above operations only consider single-head self-attention mechanisms, but these operations can be easily extended to multi-head self-attention mechanisms, thereby giving typical action network modules stronger representation capabilities.

[0118] 4. Simultaneously, this embodiment comprehensively considers the node's own position code, the position codes of neighboring nodes, and the characteristics of the edges connecting neighboring nodes to update the position code p of each node. i The update process is as follows

[0119]

[0120] Here, Γ(·) represents the information propagation process similar to node features, except that only the positional encoding of the node is considered during the calculation, and the feature information of the node is not considered.

[0121] Example 2: To verify the effectiveness of the temporal action localization method based on a typical action network as described in Example 1, this example is designed to verify the above method. The specific process includes...

[0122] Step 1: Building the dataset

[0123] This embodiment uses two datasets for experiments: THUMOS14 and ActivityNet-v1.3. The THUMOS14 dataset is from http: / / crcv.ucf.edu / THUMOS14 / , and the ActivityNet-v1.3 dataset is from http: / / activity-net.org / . The THUMOS14 training dataset contains 20 action categories, while the ActivityNet-v1.3 training dataset contains 200 action categories. Each training video contains multiple action instances. Figure 2 As shown, the method described in this embodiment is used to process all videos individually;

[0124] Step 2: Extract video features

[0125] Given a video, the I3D model is used to extract a series of video features from the video to obtain a high-level semantic representation of the video:

[0126] F = [f1, f2, ..., f T ],

[0127] in, Let f represent the t-th video feature. t It reflects information from a continuous video frame; the I3D model was proposed by Joao Carreira et al. in their 2017 work, Quo vadis, action recognition a new model and the kinetics dataset [C] / / proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:6299-6308.

[0128] The I3D model was pre-trained on the Kinetics-400 dataset, which is available at: https: / / deepmind.com / research / open-source / kinetics;

[0129] Step 3: Construct a typical action network module

[0130] This embodiment uses Deep Graph Library 0.8.2 to implement a typical action network module. In the typical action network module, typical actions are run in the form of a graph transformer network. Specifically:

[0131] Step 3.1: First, initialize the node and edge features of the graph transformation network, and use a linear transformation layer to transform typical action features. Projected into the hidden space In addition, consider a path consisting of node v i Pointing to node v j The edge is represented by the characteristic e. ij ;

[0132] Step 3.2: In the initialization phase, concatenate x i and x j The features of the two nodes are analyzed, and a linear transformation layer is applied to project these features. Furthermore, a random walk strategy is employed to generate a positional code for each node, and a linear transformation layer is used to project the positional code into the latent space.

[0133] Step 3.3: A typical action network module uses a graph transformation network for each layer. For the graph transformation network of the l-th layer, the feature of its i-th node is: The corresponding position code is Connect node v i and v j The features corresponding to the edges are Using projection matrix To process node features and their corresponding location encodings, a projection matrix is ​​used. To process edge features;

[0134] For computing node v i and its neighboring node v j , j∈N i Correlation, estimate the correlation vector

[0135]

[0136] in, [·] indicates element-wise multiplication, and [·] indicates concatenation along the channel dimension;

[0137] Step 3.4: Then, calculate the similarity in the following form.

[0138]

[0139] Update node v according to the following formula i Features

[0140]

[0141] Step 3.5: Finally, standard operations in graph transformation networks are used to process node features, sequentially using residual connections, layer normalization, and feed forward networks. To simplify the description, the above operations only consider single-head self-attention mechanisms, but these operations can be easily extended to multi-head self-attention mechanisms, thereby giving typical action network modules stronger representation capabilities.

[0142] Meanwhile, this embodiment comprehensively considers the node's own position code, the position codes of neighboring nodes, and the characteristics of the edges connecting neighboring nodes to update the position code p of each node. i ;p i The update process can be represented as

[0143]

[0144] Wherein, Γ(·) represents the information propagation process similar to node features, except that only the position encoding of the node is considered during the calculation, and the feature information of the node is not considered.

[0145] Step 4: Combine typical action network modules and ActionFormer

[0146] Step 4.1: First, the typical action network module is introduced into the temporal action localization method ActionFormer. In the interaction with typical actions, this method can provide complete typical action information for each frame, which helps to perform action modeling.

[0147] Step 4.2: Then, use max pooling to downsample the features to obtain the same number of representative features as ActionFormer;

[0148] Step 4.2: Next, using enhanced video features, and referencing ActionFormer, the temporal action localization task is solved in a one-stage, anchor-free manner. Specifically:

[0149] (1) For action category classification tasks, this method collects video frame features from typical action network modules of all categories to obtain feature representations. L2 regularization is applied to all features, then the dot product similarity between video frame features and canonical features is calculated, and the average of the similarity scores for class c is taken to obtain the classification score s. c Finally, the classification score s of the video frame is obtained;

[0150] (2) For the action boundary regression task, this method uses a 3-layer 1D convolutional neural network to predict the distance from the current time step to the boundary of the action. and This represents the distance from the current moment to the start of the action. This represents the distance from the current moment to the moment the action ends;

[0151] The kernel size is set to 3, the first two layers use layer normalization, and all three layers use the ReLU activation function.

[0152] The typical action network module proposed in this method does not require modification of the existing ActionFormer method's network structure. Instead, it constructs an additional branch, providing plug-and-play service to existing methods and significantly improving their performance in temporal action localization tasks. The temporal action localization results obtained are as follows: Figure 3 As shown;

[0153] Step 5: Training and Testing

[0154] During the training phase, the loss function is calculated in the following form.

[0155] L = L van +L exp +λ·L cons

[0156] Among them, L van This represents the loss term calculated by the ActionFormer algorithm, and the specific calculation method is as follows:

[0157]

[0158] This refers to Focalloss, proposed by Tsung-Yi Lin et al. in their 2017 work, Lin TY, Goyal P, Girshick R, et al. Focal loss for dense object detection [C] / / Proceedings of the IEEE international conference on computer vision. 2017:2980-2988., which is used for action category classification. This refers to the DIoU loss proposed by Zhaohui Zheng et al. in their 2020 work, Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression[C] / / Proceedings of the AAAI conference on artificial intelligence.2020,34(07):12993-13000., used for regression of action boundaries; T + This indicates the number of positive samples in a video. It is an indicator function used to indicate whether the current moment belongs to a certain action; λ reg Represents the equilibrium coefficient, used for balancing. and

[0159] Given the prediction results of typical action branches, the corresponding loss term L for each typical branch can be calculated using the ActionFormer algorithm. exp The ActionFormer algorithm's prediction results are kept consistent with those of typical action branches. The consistency loss term L is calculated using the mean squared error function. cons λ is used to adjust for the impact of consistency loss;

[0160] During the testing phase, the prediction results of the ActionFormer algorithm were fused using hyperparameter β. van Prediction results of typical action branches exp The results of the temporal action localization task are obtained.

[0161] s = β·s van +(1-β)·s exp .

[0162] This method uses the Soft-NMS method proposed by Navaneeth Bodla et al. in their 2017 work, Bodla N, Singh B, Chellappa R, et al. Soft-NMS--improving object detection with one line of code[C] / / Proceedings of the IEEE international conference on computer vision.2017:5561-5569., to post-process the temporal action localization results, remove highly overlapping action instances, and produce the final result of temporal action localization.

[0163] Step 6: Parameter settings.

[0164] This method references Devin Kreuzer et al.'s 2021 work, Kreuzer D, Beaini D, Hamilton W, et al. Rethinking graph transformers with spectral attention[J]. Advances in Neural Information Processing Systems, 2021, 34: 21618-21629, and Vijay Prakash Dwivedi et al.'s 2022 work, Dwivedi VP, Luu AT, Laurent T, et al. Graph neural networks with learnable structural and positional representations[J]. arXiv preprint arXiv:2110.07875, 2021. The relevant parameters of the graph transformation network layer are set, and the dimensions of node features and positional encoding features are set to 64.

[0165] In addition, the hyperparameters of this method are adjusted according to the experimental performance of the validation set; finally, the typical number M is set to 10, the typical number of action layers is set to 3, the loss coefficient λ is set to 1.0, and the fusion coefficient β is set to 0.5.

[0166] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A temporal action localization method based on a typical action network, characterized in that: include Step 1: Extract video features from the given video; Step 2: Generate typical actions using a clustering algorithm based on video features; Step 2 describes the process of generating typical actions using clustering algorithms, which includes... Step 2.1: Consider a class of actions, collect the feature representations of all action instances, and use the K-Means algorithm to cluster the features to obtain M clusters; for each cluster feature representation, take the feature closest to the cluster center as the typical feature to obtain M typical features; Step 2.2: After obtaining M typical features, divide each action instance evenly into three stages: action beginning, action evolution, and action ending; Step 3: Based on the information transmission relationships between typical actions, construct typical action network modules based on structural information transmission; Step 3 involves constructing a typical action network module. Step 3.1: Connect multiple typical actions belonging to the same stage, and perform bidirectional information exchange between every two typical actions; Step 3.2: Information transfer from "action start" to "action evolution" and from "action evolution" to "action end"; connect each typical feature of the start stage with all typical features of the evolution stage, and connect each typical feature of the evolution stage with all typical features of the end stage, and establish a directed graph between typical features of different stages. Step 3.3: Aggregate typical information into video frames; at the same time, compare the video frame features with all typical features one by one, and use an attention mechanism to adaptively aggregate features from each action stage; Step 3.4: Use video frame broadcasting to pass the characteristics of the current video frame to the typical action network module; each typical action takes into account its own characteristics and the characteristics of the video frame being processed, and updates its own characteristics according to the attention mechanism; Step 4: Combine the typical action network module with the actionformer temporal action localization method; Step 5: Perform classification estimation based on similarity comparison on the results of combining typical action network modules with the temporal action localization method ActionFormer; Step 6: Set the loss function and train and test the classification results.

2. The temporal action localization method based on a typical action network according to claim 1, characterized in that: The process of extracting video features from a given video as described in step 1 is as follows: Given a video, the I3D model is used to extract a series of video features from the video to obtain a high-level semantic representation of the video: ， in, Indicates the first Video features, It reflects information from a series of video frames.

3. The temporal action localization method based on a typical action network according to claim 2, characterized in that: In the typical action network module described in step 3, the typical action operates in the form of a graph transformation network, and its specific application process is as follows: First, initialize the node and edge features of the graph transformation network. (1) Use a linear transformation layer to transform typical action features Projected into the hidden space Furthermore, consider a path consisting of nodes Pointing to node The edges are represented as follows: ; (2) During the initialization phase, splicing and The features of the two nodes are analyzed, and a linear transformation layer is applied to project these features. ; In addition, a random walk strategy is used to generate a position code for each node. And a linear transformation layer is used to project the position code into the latent space. ; (3) A typical action network module uses a graph transformation network for each layer. For the first layer, ... The graph transformation network of the first layer, its first layer The characteristics of each node are: The corresponding position code is Connecting nodes and The features corresponding to the edges are Using projection matrix To process node features and their corresponding location encodings, a projection matrix is ​​used. To process edge features; (4) is a computing node and its neighboring nodes The correlation is estimated, and the correlation vector is: ; ； in, This indicates element-wise multiplication. This indicates splicing along the channel dimension; Similarity is calculated using the following formula. ； Update nodes according to the following formula Features: ； The standard operations in graph transformation networks are used to process node features, followed by residual connections, layer-by-layer regularization, and feedforward networks. Update the position code of each node by considering its own position code, the position codes of its neighboring nodes, and the features of the edges connecting neighboring nodes. The update process is as follows ； in, This means that during the propagation of information similar to node features, only the node's positional encoding is considered in the calculation, without considering the node's feature information.

4. The temporal action localization method based on a typical action network according to claim 1, characterized in that: Step 4 describes the process of combining a typical action network module with the ActionFormer temporal action localization method. Step 4.1: Introduce the typical action network module into ActionFormer, and build an additional branch in the network structure of the ActionFormer method; Step 4.2: Use max pooling to downsample the features to obtain the same number of representative features as ActionFormer; Step 4.3: Using enhanced video features, referencing ActionFormer, solve the temporal action localization task in a one-stage, anchor-free manner.

5. The temporal action localization method based on a typical action network according to claim 1, characterized in that: Step 5 describes the classification estimation process based on similarity comparison, which includes... Step 5.1: In a typical action network module, the node features of the last layer graph transform network are represented as follows: ； in, Representing the kind Characteristics of a typical node Representative by the first The video frame features represented by the class; video frame features are collected from typical action network modules of all classes to obtain feature representations. ; Step 5.2: For the action category classification task, collect video frame features from typical action network modules of all categories to obtain feature representations. To obtain the predicted classification score, all features are processed. Regularization is performed, and then the dot product similarity between the video frame features and the canonical features is calculated. The classification score is obtained by averaging the similarity scores of the classes. Finally, the classification scores of the video frames are obtained. ; For the action boundary regression task, a 3-layer 1D convolutional neural network is used to predict the distance from the current time step to the boundary of the action. and ; in, This represents the distance from the current moment to the start of the action. It represents the distance from the current moment to the moment the action ends.

6. The temporal action localization method based on a typical action network according to claim 1, characterized in that: The training and testing process described in step 6 includes Step 6.1: During the training phase, let the loss function be as follows: ； in, This represents the loss term calculated by the ActionFormer algorithm, and the specific calculation method is as follows: ； This represents Focal loss, used for classifying action categories; This represents DIoU loss, used for regression of action boundaries; This indicates the number of positive samples in a video. It is an indicator function used to indicate whether the current moment belongs to a certain action; Represents the equilibrium coefficient, used for balancing. and ; Given the prediction results of typical action branches, calculate the corresponding loss terms for typical branches using the ActionFormer algorithm. The ActionFormer algorithm's predictions are kept consistent with those of typical action branches; a consistency loss term is calculated. , Used to adjust for the impact of consistency loss; Step 6.2: During the testing phase, use hyperparameters. Prediction results from the ActionFormer algorithm Prediction results of typical action branches The final result of the relevant video interpretation task is as follows: 。

Images