A graph network model for temporal action detection

Abstract

The application relates to a graph network model for timing action detection and belongs to the technical field of timing action detection tasks. In a proposal generation stage, action proposals are obtained through a basic feature network and a proposal generation network; after the action proposals are obtained, specific action labels of the action proposals are predicted in a proposal classification stage; in the stage, a proposal classification network has a cross-scale proposal graph module and a semantic proposal graph module to obtain high-quality proposal features; in the cross-scale proposal graph module, a graph attention convolution network A of a cross-time feature pyramid is designed to generate proposal features M with different granularity time context information; in the semantic proposal graph module, the proposal features M with different granularity time context information are constructed into a proposal graph, semantic information is transmitted between the proposals by using a graph attention convolution network B, enhanced proposal features N are obtained, and finally the enhanced proposal features N are provided to a classifier to predict the labels of the proposals.

Description

TECHNICAL FIELD

[0001] The present application relates to a graph network model for temporal action detection, and belongs to the technical field of temporal action detection task. BACKGROUND

[0002] Temporal action detection aims to simultaneously recognize the temporal boundaries and categories of all actions in an untrimmed video, and has attracted great attention in the research community. Similar to object detection, most temporal action detection methods adopt a two-step approach, first generating class-agnostic temporal action proposals, and then predicting their action labels. Some methods treat these two steps as independent sub-tasks and design different deep ConvNets for each, however, these methods train each stage separately, leading to sub-optimal computational efficiency and lower performance.

[0003] Some methods share the convolutional features between the proposal generation and classification stages to build an end-to-end model, and train them by jointly optimizing the proposal generation and classification stages. Action proposals usually contain partial ground truth, lacking enough semantic information to recognize incomplete actions, and obtaining information from other proposals can gain a lot of valuable information, because irrelevant proposals have been removed when generating action proposals. However, most end-to-end models predict the labels of proposals separately and ignore the relationships between proposals.

[0004] Still some methods build a temporal feature pyramid to address the inherent temporal scale variability of action instances, however, these end-to-end models classify proposals at a single level of the temporal feature pyramid, ignoring the information transfer between proposal features at different temporal scales.

[0005] To obtain the cross-scale correlation and semantic relationship between different proposals and improve the performance of temporal action detection, it is necessary to enhance the proposal features by predicting their specific action labels at the proposal classification stage. A video contains hundreds of proposals, most of which contain redundant or irrelevant information to each other, and it is very important to obtain valuable information from other proposals to improve the quality of proposal features. SUMMARY

[0006] In view of this, the present application proposes a graph network model for timing action detection, called cross-scale semantic proposal graph network (CSSPGN), which explores the cross-scale correlation of the same action proposal and the semantic relationship of different action proposals to improve the proposal features. In the proposal generation stage, action proposals are obtained through the basic feature network (BFN) and the proposal generation network (PGN); after obtaining the action proposals, their specific action labels are predicted in the proposal classification stage, in which the proposal classification network (PCN) has two key components, namely the cross-scale proposal graph (CSPG) module and the semantic proposal graph (SPG) module, to obtain high-quality proposal features. In the cross-scale proposal graph module, a graph attention convolutional network A of a cross-time feature pyramid is designed to generate proposal features M with different granularity of temporal context information, and in the semantic proposal graph module, the proposal features M with different granularity of temporal context information are constructed into a proposal graph, and a graph attention convolutional network B is used to transmit semantic information between proposals to obtain enhanced proposal features N, and finally the enhanced proposal features N are provided to the classifier to predict the labels of the proposals.

[0007] The technical solution of the present application is:

[0008] A graph network model for timing action detection, the graph network model comprising a basic feature network, a proposal generation network and a proposal classification network, the proposal classification network comprising a cross-scale proposal graph module and a semantic proposal graph module.

[0009] The basic feature network detects action instances at various time scales according to the video feature sequence extracted from the original video data, and is given a video feature sequence f∈R T×C , where T is the length of the video feature sequence and C is the dimension of the video feature dimension, and the basic feature network first embeds it into a D-dimensional space using a convolutional network. The convolutional network is composed of L1D convolutional layers with ReLU as the activation function

[0010] f u =ConvNet(f)(1)

[0011] where f u ∈R T×D is the output of the convolutional network, then the basic feature network performs N-level downsampling operation on f u to obtain a time feature pyramid , where is the i-th feature map of the time feature pyramid.

[0012] The proposal generation network generates proposals using an anchor-free mechanism, and selects the i-th feature map of the time feature pyramid The classification and regression branch predicts the confidence score p j and the boundary distance of each position j Find candidate proposals:

[0013]

[0014] According to the confidence score, the proposal generation network selects N p candidate proposals as action proposals and inputs them into the proposal classification network, where t s,j , t e,j are the start and end times of the candidate proposal ψ j .

[0015] The cross-scale proposal graph module of the proposal classification network designs a graph attention convolution network A of a cross-time feature pyramid to generate proposal features M with different granularity of temporal context information, and the cross-scale proposal graph module includes the following steps:

[0016] Step 1: receiving the action proposals generated by the i-th level of the time feature pyramid output by the proposal generation network using the ROI pooling layer to obtain the proposal features across the time scale, obtaining N+1 proposal features across the time scale wherein is the proposal feature generated from the k-th level of the time feature pyramid;

[0017] Step 2: constructing the N+1 proposal features across the time scale obtained in step 1 into a graph G j ={V j ,E j}, wherein represents a node, and E j represents an edge set;

[0018] Step 3: using the graph attention convolution network A to obtain the temporal context information on the graph G j ={V j ,E j} generated in step 2, which gives adjacent nodes weights to represent their importance, and the graph attention convolution module is written as follows:

[0019]

[0020] wherein W is a parameter matrix to be learned, N(v) is the entire adjacent node set of the v-th node, a vq is the weight of the q-th adjacent node of the v-th node, h q represents the feature of the q-th adjacent node, and the weight is calculated using the attention mechanism:

[0021]

[0022] W aare learnable parameters; || is a repeated serial operation, in the output of the graph attention convolution module, to obtain proposal features M with different granularity temporal context information and output them to the semantic proposal graph module.

[0023] The semantic proposal graph module of the proposal classification network is used to construct different proposals in the video into a proposal graph, and pass semantic information between proposals using the graph attention convolution network B to obtain enhanced proposal features N, and finally provide the enhanced proposal features N to the classifier to predict the label of the proposal. The semantic proposal graph module includes the following steps:

[0024] Step 1: Construct a proposal graph according to the proposal features M with different granularity temporal context information received from the semantic proposal graph module, denoted as G p = {V p , E p}, where, is a set of nodes, E p is a set of edges, and the adjacency matrix is defined as is used to measure the relevance between nodes; the relevance between node and other nodes is calculated and the K nodes with the highest relevance are selected to create edges, so the edges of node are represented as follows:

[0025]

[0026] where ArgTopk() represents the index of the largest K elements of the input vector, is the adjacent node of ;

[0027] Step 2: Generate a confidence score for each proposal in the proposal graph constructed in step 1 to represent the possibility of containing an action instance, and use the confidence to delete low-quality adjacent nodes of each proposal, by obtain the K adjacent nodes of node , compare their confidence scores with the threshold S thr , once the confidence score is higher than S thr , create an edge between the adjacent node and node ;

[0028] Step 3: After deleting the low-quality adjacent nodes of each proposal in step 2, use the M-layer graph attention convolution network B to obtain action information from the adjacent nodes of each proposal to enhance the proposal features, and the m-th layer of the graph attention convolution layer is written as:

[0029]

[0030] where N'(v) represents the node a set of all neighboring nodes of the vth proposal, cosine similarity of the proposed features as the attention weight of the vth proposal and the jth neighboring node

[0031]

[0032] generating the proposed features N enhanced by the graph attention convolutional network B;

[0033] Step 4: sending the proposed features N enhanced by the graph attention convolutional network B in step 3 into a fully connected (FC) layer with a softmax operation as an action classifier, and using n binary classifiers as integrity classifiers, each corresponding to an action class, to predict the label of the proposal.

[0034] A method for temporal action detection using the above-mentioned graph network model, the steps comprising:

[0035] First, the basic feature network extracts a video feature sequence from the original video data, and constructs a time feature pyramid based on the extracted video feature sequence using a transformer to detect action instances at various time scales, and inputs the constructed time feature pyramid into the proposal generation network;

[0036] Second, after receiving the time feature pyramid output by the basic feature network, the proposal generation network finds candidate proposals according to the i-th feature map of the received time feature pyramid, the confidence score, and the time distance from each position in the i-th feature map to the boundary, selects N p candidate proposals from the found candidate proposals as action proposals, and inputs the selected action proposals into the cross-scale proposal graph module in the proposal classification network;

[0037] Third, the cross-scale proposal graph module in the proposal classification network receives the action proposals output by the proposal generation network, generates proposal features M with different granularity temporal context information through the graph attention convolutional network A, and outputs the proposal features M to the semantic proposal graph module;

[0038] Fourth, the semantic proposal graph module in the proposal classification network receives the proposal features M with different granularity temporal context information generated by the cross-scale semantic graph module, constructs a proposal graph according to the received proposal features M, and applies the graph attention convolutional network B to the constructed proposal graph to obtain enhanced proposal features N, and finally sends the enhanced proposal features N into an action classifier to predict the label of the proposal, completing the temporal action detection.

[0039] Advantages

[0040] The application is a graph network model for temporal action detection, which has excellent effect in the temporal action detection task, and the performance of the model on the THUMOS'14 dataset has been significantly improved compared with the current most advanced method, which proves the effectiveness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A cross-scale semantic proposal graph network structure is provided.

[0042] Figure 2 A cross-scale proposal graph module framework is provided.

[0043] Figure 3 A semantic proposal graph module framework is provided. DETAILED DESCRIPTION

[0044] The application will be further described below in combination with the drawings and examples.

[0045] The application proposes a new cross-scale semantic proposal graph network to accurately locate action instances in untrimmed videos. Considering that an untrimmed video V contains N g action instances Φ g :

[0046]

[0047] Where s gt,n ,e gt,n is the start and end time of the action instance Φ n , y gt,n is the label of the action instance Φ n . The input of the cross-scale semantic proposal graph network is the video feature sequence extracted from the original video data. In order to extract the feature sequence, the untrimmed video is divided into non-overlapping segments, and a pre-trained start action model is used to extract features from each segment.

[0048] A method for temporal action detection using a graph network model, comprising the steps of:

[0049] First, the basic feature network extracts a video feature sequence from the original video data, and constructs a time feature pyramid using a transformer according to the extracted video feature sequence to detect action instances at various time scales, and inputs the constructed time feature pyramid into the proposal generation network.

[0050] Second, the proposal generation network receives the time feature pyramid output by the basic feature network, and finds candidate proposals according to the i-th feature map of the received time feature pyramid, the confidence score, and the time distance from each position in the i-th feature map to the boundary, and selects N pThe selected action proposal is input into a cross-scale proposal graph module in the proposal classification network as an action proposal;

[0051] In the third step, the cross-scale proposal graph module in the proposal classification network receives the action proposal output by the proposal generation network, generates proposal features M with different granularity temporal context information through a graph attention convolution network A, and outputs the proposal features M to a semantic proposal graph module;

[0052] In the fourth step, the semantic proposal graph module in the proposal classification network receives the proposal features M with different granularity temporal context information generated by the cross-scale semantic graph module, constructs a proposal graph according to the received proposal features M, and applies a graph attention convolution network B to the constructed proposal graph to obtain enhanced proposal features N. Finally, the enhanced proposal features N are sent to an action classifier to predict the label of the proposal, and the temporal action detection is completed.

[0053] As shown in Figure 1 , the cross-scale semantic proposal graph network is composed of a basic feature network, a proposal generation network and a proposal classification network. The basic feature network uses a transformer block with local self-attention to create a multi-scale temporal feature pyramid to capture action instances at various temporal scales; the proposal generation network uses an anchor-free method and is applied to each layer of the temporal feature pyramid to generate proposals; the proposal classification network is the most significant contribution of the present application, which explores the cross-scale correlation and semantic relationship between proposals to improve the performance of temporal action detection.

[0054] The following are the three components of the cross-scale semantic proposal graph network, i.e. its three subnetworks:

[0055] 1.1 Basic feature network

[0056] In order to detect action instances at various temporal scales, the basic feature network uses transformer to construct a temporal feature pyramid. Transformer is a self-attention mechanism that can capture long-term correlations in sequences. However, the high computational cost of transformer limits its application in long sequences. In order to reduce the computational cost of transformer, the self-attention layer of transformer is replaced by a local self-attention layer, which limits the attention to a local window ω. Given a video feature sequence f∈R T×C , the basic feature network first embeds it into a D-dimensional space using a convolutional network consisting of L1D convolutional layers with ReLU activation function, defined as follows:

[0057] f u =ConvNet(f)(9)

[0058] where f u∈ R T×D is the output of the convolutional network. Then, the base feature network performs N-level down-sampling operations on f u to obtain the temporal feature pyramid where is the i-th level feature map of the temporal feature pyramid.

[0059] 1.2 Proposal generation network

[0060] The proposal generation network generates proposals with an anchor-free mechanism, which directly predicts the confidence score and the temporal distance from each location in the feature map to the boundary without pre-defining anchors. The proposal generation network consists of a classification and regression branch, and all levels share parameters. The classification branch consists of 3 ID convolutional layers with kernel size = 3. The first two layers use ReLU as the activation function, and the last layer uses the sigmoid function as the activation function. The regression branch uses the same design as the classification branch, except that the last layer uses ReLU as the activation function for distance estimation.

[0061] For the i-th level feature map of the temporal feature pyramid The classification and regression branch predicts the confidence score p j and the boundary distance d from each location j.

[0062]

[0063] where t s,j , t e,j are the start and end times of the candidate action proposal ψ j The proposal generation network uses a soft NMS method among all candidate proposals at each level of the temporal feature pyramid to eliminate redundant proposals. According to the confidence score, the proposal generation network selects N p candidate proposals as the action proposal and inputs it into the proposal classification network.

[0064] 1.3 Proposal classification network

[0065] The proposal classification network is designed to predict the class label of the action proposal. The proposal classification network designs a cross-scale proposal graph module to generate proposal features M with different granularities of temporal context information. This module explores the cross-scale relationship of the same proposal to obtain multi-granularity information, and constructs a semantic proposal graph module to improve the quality of the proposal features. This module uses the semantic relationship of different proposals to obtain complementary information. Finally, the proposal classification network provides the proposal features N enhanced by the graph attention convolutional network B to the classifier to predict the label of the proposal.

[0066] The following is the cross-scale proposal graph module and the semantic proposal graph module for proposal generation and proposal classification respectively by the cross-scale semantic proposal graph network:

[0067] 1.3.1 Cross-scale proposal graph module

[0068] As Figure 2 The overall framework of the cross-scale proposal graph module is shown. For the action proposal generated by the i-th level of the temporal feature pyramid output by the proposal generation network The proposal features across the temporal scale are obtained using the ROI pooling layer.

[0069] Given the proposal ψ j generated from the i-th level of the temporal feature pyramid N+1 proposal features across the temporal scale can be obtained where ψ j is the proposal feature generated from the k-th level of the temporal feature pyramid. The proposal features are constructed into a proposal graph, and a graph attention convolutional network A is used to obtain different granularity of temporal context information. The graph can be represented as G j = {V j , E j}, where V represents the node, and E vq represents the edge set. In the graph, the node V represents the proposal feature generated from the b-th level of the temporal feature pyramid The edge set is defined as follows:

[0070]

[0071] Different granularities of temporal context information have different contributions to the classification of proposals. Therefore, a graph attention convolutional module is applied to obtain temporal context information on the graph, thereby assigning weights to adjacent nodes to indicate their importance. The graph attention convolutional module can be written as follows:

[0072]

[0073] where W is the parameter matrix to be learned, N(v) is the entire adjacent node set of the v-th node, a vq is the weight of the q-th adjacent node of the v-th node, h q represents the feature of the q-th adjacent node, and the weight is calculated using the attention mechanism:

[0074]

[0075] where W a is a learnable parameter, and || is a repeated concatenation operation. In the output of the graph attention convolutional module, the proposal feature M with different granularity of temporal context information is obtained and output to the semantic proposal graph module.

[0076] 1.3.2 Semantic Proposal Graph Module

[0077] As Figure 3 The semantic proposal graph module constructs a proposal graph G according to the proposal features M with different granularity temporal context information received from the cross-scale semantic graph module, and applies a graph attention convolution network B to obtain enhanced proposal features N, which represents the semantic proposal graph module as p = {V p ,E p}, where, represents a set of nodes, E p is a set of edges. The adjacency matrix is defined as is used to measure the relevance of nodes, and the relevance is calculated between the node and other nodes and the K nodes with the highest relevance are selected to create edges. Therefore, the edges of the node are represented as follows:

[0078]

[0079] where ArgTopk() represents the index of the largest K elements of the input vector, is the adjacent node of ;

[0080] A time position vector is constructed to represent the time position relationship of the proposal, which can be represented as:

[0081] s vj = [t v,s ,t v,e ,l p,v ,t j,s ,t j,e ,l p,j ] (15)

[0082] where s vj is the time position vector of the proposal p v and p j , t v,s and t v,e are the start and end coordinates of p v , respectively, and l p,v and l p,j are the lengths of the proposals p v and p j . The adjacency matrix A s is calculated as follows:

[0083]

[0084] where It is the adjacency matrix A s One element, W1 and W2 are learnable parameters, B ias It is a learnable bias.

[0085] Because proposals are likely to contain action information, a proposal graph is constructed and a graph attention convolutional network B is used to obtain action information from other proposals. The proposal generation network generates a confidence score for each proposal to represent the probability of containing an action instance. The confidence score is used to remove low-quality neighbors for each proposal. Obtain Node After finding its K neighboring nodes, its confidence score is compared with the threshold S. thr For comparison, once the confidence score is higher than S thr In adjacent nodes and nodes Create edges between them.

[0086] After obtaining the graph, an M-layer graph attention convolutional network is used to obtain action information from the neighboring nodes of each proposal. The m-th graph attention convolutional layer can be written as:

[0087]

[0088] Where N'(v) represents the node Given the entire set of neighboring nodes, the cosine similarity of the proposal features is used as the attention weight between the v-th proposal and the j-th neighboring node.

[0089]

[0090] The proposal features N, enhanced by a graph attention convolutional network B, are fed into an action classifier to predict the proposal's label. The action classifier, trained to predict the category of video segments, typically does not affect completeness. An completeness classifier is added to filter out proposals containing incomplete action instances. A fully connected (FC) layer with a softmax operation is used as the action classifier, and n binary classifiers are used as completeness classifiers, each corresponding to an action class, to predict the proposal's label.

[0091] The following describes the training process of the network and the results obtained from experiments on two standard datasets:

[0092] To train the proposal classification network, the temporal location information of the temporal feature pyramid in the ground truth action instance is treated as a positive sample, and the others are treated as negative samples, in order to optimize the classification branch.

[0093] The temporal distance between the boundary and the positive position is calculated to optimize the regression branch. After obtaining the proposal, it is divided into three categories by evaluating its temporal intersection-union ratio (tIoU) with the basic fact action instances:

[0094] (1) Positive proposals with tIoU greater than 0.7 have the closest action instance to the basic fact;

[0095] (2) Background proposal, tIoU less than 0.01, with the action instance closest to the ground truth;

[0096] (3) Incomplete proposals with tIoU below 0.3 but greater than 0;

[0097] The action classifier is trained using positive and background proposals, while the completeness classifier is trained using positive and incomplete proposals.

[0098] To train the proposal generation and classification networks, it is necessary to jointly optimize the loss function L of the proposal generation and classification networks. pgn and L pcn Proposed Generative Network L pgn The loss function has two parts: binary classification loss L b and regression loss L reg For the proposal classification network, an action classification loss L is integrated. c And an integrity loss L com To train. Total loss L total Defined as:

[0099]

[0100] Where λ1 and λ2 are balance coefficients, L b It's a focal length loss, L reg For generalized IoU loss, L c It is the cross-entropy loss, L com It is the hinge loss, used for each binary classifier of the integrity classifier.

[0101] In this invention, candidate proposals are first found at each time position of the time feature pyramid, and redundant proposals are removed using the soft NMS method to obtain action proposals. Then, these action proposals are submitted to a proposal classification network to predict their category and integrity scores. Finally, the category score is multiplied by the confidence score to obtain the final confidence score for each proposal.

[0102] We conduct experiments on two standard datasets, THUMOS'14 and ActivityNet-1.3, to evaluate the graph network model for temporal action detection, and the average mean average precision (mAP) of certain tIoU thresholds is used as the evaluation index. According to the official evaluation API, the average precision (mAP) is reported using multiple tIoU thresholds, and in addition, the average mAP performance of the ActivityNet-1.3 dataset is also reported, where the tIoU is set to 0.5 to 0.95 with a step of 0.05. The experimental results are shown in Tables 1 and 2:

[0103] Table 1 Temporal action detection performance (mAP %) on THUMOS'14 with different tIoU thresholds

[0104]

[0105] Table 2 Temporal action detection performance (mAP %) on ActivityNet-1.3 with different tIoU thresholds

[0106]

[0107] The experimental results show that the cross-scale semantic proposal graph network model proposed in the present application has significantly improved the performance of temporal action detection on the THUMOS'14 dataset compared to the current most advanced method, proving the effectiveness of the model.

Claims

1. A graph network model for temporal action detection, characterized in that: This graph network model includes a basic feature network, a proposal generation network, and a proposal classification network; The proposed classification network includes a cross-scale proposal graph module and a semantic proposal graph module; The basic feature network is used to extract video feature sequences from the raw video data, and to construct a temporal feature pyramid using a transformer based on the extracted video feature sequences to detect action instances at various time scales. The constructed temporal feature pyramid is then input into the proposal generation network. The proposed generation network receives the temporal feature pyramid output by the basic feature network, and finds candidate proposals based on the i-th level feature map of the received temporal feature pyramid, the confidence score, and the time distance from each position in the i-th level feature map to the boundary. It then selects a proposal from the found candidate proposals. Each candidate proposal is used as an action proposal, and the selected action proposal is input into the proposal classification network; The cross-scale proposal graph module in the proposal classification network is used to receive action proposals output by the proposal generation network, generate proposal features M with different granular temporal context information through the graph attention convolutional network A, and output the proposal features M to the semantic proposal graph module. The method steps for generating proposal features M with different granular temporal context information in the cross-scale proposal graph module of the proposal classification network include: Step 1: Receive the action proposals generated by the i-th level of the temporal feature pyramid output by the proposal generation network. The ROI pooling layer is used to obtain N+1 proposed features across time scales. ,in These are proposed features generated from the k-th level of the temporal feature pyramid. Step 2: Construct a graph from the N+1 proposed features across time scales obtained in Step 1. ,in Represents a node. Represents an edge set; Step 3: Transfer the graph generated in Step 2 The cross-temporal scale information is input into the graph attention convolutional network A to obtain proposed features M with temporal context information of different granularities; The graph attention convolutional network A is: （3） in, It is the parameter matrix to be learned. It is the set of all neighboring nodes of the v-th node. It is the weight of the q-th neighbor of the v-th node. The features of the q-th neighbor node are represented, and the weights are calculated using an attention mechanism: （4） in, It is a learnable parameter; || is a repeated concatenation operation; The semantic proposal graph module in the proposed classification network is used to receive proposal features M with different granular temporal context information generated by the cross-scale semantic graph module, construct a proposal graph based on the received proposal features M, apply a graph attention convolutional network B to the constructed proposal graph to obtain enhanced proposal features N, and finally feed the enhanced proposal features N into the action classifier to predict the label of the proposal.

2. The graph network model for temporal action detection according to claim 1, characterized in that: The basic feature network extracts video feature sequences from the original video data as follows: ; Where T is the length of the video feature sequence and C is the dimension of the video feature sequence; Constructing a time feature pyramid based on the extracted video feature sequence refers to: [The text abruptly ends here, likely due to an incomplete sentence or a missing section.] A convolutional network is embedded into a D-dimensional space to obtain the output of the convolutional network. Then, an N-level downsampling operation is performed on the output of the convolutional network to obtain the temporal feature pyramid. ,in It is the i-th level feature map of the time feature pyramid; The convolutional network described above consists of L1D convolutional layers activated by ReLU: （1） in, It is the output of the convolutional network.

3. A graph network model for temporal action detection according to claim 2, characterized in that: In the proposed generation network, the method for finding candidate proposals is as follows: Select the i-th level feature map of the time feature pyramid. Direct prediction of classification and regression branch prediction confidence scores and each position Boundary distance Find candidate proposals: （2） in, It is a candidate proposal The start and end times.

4. A graph network model for temporal action detection according to claim 3, characterized in that: The choice mentioned The method for using candidate proposals as action proposals is as follows: based on confidence scores... Propose generating network selection Each candidate proposal is used as an action proposal. And input it into the proposal classification network.

5. A graph network model for temporal action detection according to claim 4, characterized in that: In the semantic proposal graph module of the proposed classification network, the method for obtaining enhanced proposal features N using the graph attention convolutional network B is as follows: For each proposal in the constructed proposal graph, a confidence score is generated to represent the probability that each proposal contains an action instance. Low-quality neighboring nodes of each proposal are removed. After removing the low-quality neighboring nodes of each proposal, an M-layer graph attention convolutional network B is used to obtain action information from the neighboring nodes of each proposal to obtain proposal features N.

6. A graph network model for temporal action detection according to claim 5, characterized in that: The proposed graph of the construction is represented as follows: ,in, It represents a set of nodes. It is an edge set, and the adjacency matrix is ​​defined as follows: Used to measure the correlation of nodes, at nodes Calculate the correlation between the nodes. And select the one with the highest relevance. Nodes are used to create edges, therefore, nodes The edges are represented as follows: （5） in, This represents the indices of the K largest elements in the input vector. for The adjacent nodes.

7. A graph network model for temporal action detection according to claim 6, characterized in that: Each proposal in the proposal graph generates a confidence score to represent the probability that each proposal contains an action instance, and the method for removing low-quality neighboring nodes of each proposal is as follows: Use confidence scores to remove low-quality neighbors for each proposal, through the adjacency matrix. Obtain Node of After each neighboring node, its confidence score is compared with the threshold. For comparison, once the confidence score is higher than In adjacent nodes and nodes Create edges between them.

8. A graph network model for temporal action detection according to claim 7, characterized in that: The m-th graph attention convolutional layer of the M-layer graph attention convolutional network B is: （6） in, Represents a node Given the entire set of neighboring nodes, the cosine similarity of the proposal features is used as the attention weight between the v-th proposal and the j-th neighboring node. : （7） Generate proposed features N enhanced by a graph attention convolutional network B; The proposed features N, enhanced by the graph attention convolutional network B, are fed into a fully connected layer with a softmax operation as an action classifier. n binary classifiers are used as integrity classifiers, each corresponding to an action class, to predict the proposed label and complete the temporal action detection.

9. A method for temporal action detection using the graph network model according to any one of claims 1-8, characterized in that the steps are... include: The first step is to extract video feature sequences from the original video data using the basic feature network, and construct a temporal feature pyramid using a transformer based on the extracted video feature sequences to detect action instances at various time scales. The constructed temporal feature pyramid is then input into the proposal generation network. The second step involves the proposal generation network receiving the temporal feature pyramid output by the basic feature network. Based on the i-th level feature map of the received temporal feature pyramid, the confidence score, and the time distance from each position in the i-th level feature map to the boundary, candidate proposals are found. From these candidate proposals, a selection is made. Each candidate proposal is used as an action proposal, and the selected action proposal is input into the cross-scale proposal graph module in the proposal classification network; The third step is that the cross-scale proposal graph module in the proposal classification network receives the action proposals output by the proposal generation network, generates proposal features M with different granular temporal context information through the graph attention convolutional network A, and outputs the proposal features M to the semantic proposal graph module. In the fourth step, the semantic proposal graph module in the proposal classification network receives proposal features M with different granular temporal context information generated by the cross-scale semantic graph module. It constructs a proposal graph based on the received proposal features M, and applies a graph attention convolutional network B to the constructed proposal graph to obtain enhanced proposal features N. Finally, the enhanced proposal features N are fed into the action classifier to predict the label of the proposal and complete the temporal action detection.

Images