A temporal action detection method of a temporal-visual proposal graph network

Abstract

The present application relates to a kind of time sequence-visual proposal graph network's time sequence action detection method, belong to time sequence action detection technique.This method uses time sequence graph convolution network and visual graph convolution network in parallel on proposal graph: first, for each proposal selection valuable neighbor proposal, and it is built as action proposal graph, then adopt graph convolution network according to the different contribution of neighbor proposal to classification from neighbor proposal to obtain action information to construct a visual graph convolution network, using the cosine similarity of proposal feature as weight;Again construct a time sequence graph convolution network in parallel with visual graph convolution network on action proposal graph, information is aggregated by the time sequence and visual correlation between proposals.In using graph convolution network, a new time sequence graph convolution operation is proposed, which divides neighbor proposals into multiple sub-neighbor according to the time sequence position relationship, and extracts information in different ways, uses a measurement function as auxiliary supervision to obtain more valuable information from neighbor proposals.

Description

TECHNICAL FIELD

[0001] The application relates to a time sequence action detection method of a time sequence-vision proposal graph network and belongs to the technical field of time sequence action detection. BACKGROUND

[0002] In the time sequence action detection task, most algorithms are divided into two stages of action proposal generation and proposal classification. The purpose of the time sequence action proposal generation stage is to find time sequence regions that may contain foreground actions, and these time sequence regions are called action proposals. The purpose of the proposal classification stage is to predict the action label of each proposal. High-quality action proposals are the key to time sequence action detection.

[0003] Action proposal classification is generally considered an action recognition task, and existing algorithms usually only contain part of the ground truth action when generating proposals, lacking sufficient semantic information to accurately predict their categories. Since a video usually has multiple action instances, if each proposal obtains one information, a large amount of noise will be introduced, and the use of proposal feature similarity as a weight will also lead to more redundant information and less valuable information. Ignoring the time sequence position relationship between proposals will also limit the detection performance. SUMMARY

[0004] The technical problem to be solved by the application is to overcome the shortcomings of the prior art and provide a time sequence action detection method of a time sequence-vision proposal graph network (TVPGN).

[0005] The technical solution of the application is:

[0006] A time sequence action detection method of a time sequence-vision proposal graph network, the steps of the method comprising:

[0007] Step 1: constructing a proposal graph in the uncropped video V, the specific method being:

[0008] performing an operation on the generated proposal set Ψ p of the uncropped video V, and Ψ p is expressed as:

[0009]

[0010] In the formula, N p is the number of generated proposals in the video V, c p,n , and l p,n are the center coordinates and length of the generated proposal p n , respectively. The steps of the operation are as follows:

[0011] First, for each proposal p p in Ψ n , first remove the proposal p p from Ψn The proposals other than the 10000 proposals are divided into different categories. The classification method is as follows: Based on the temporal position relationship, each proposal has five neighbor proposals:

[0012] 1) The center coordinate of the neighbor proposal is smaller than the center coordinate of the proposal on the time axis, and the two overlap;

[0013] 2) The center coordinate of the neighbor proposal is smaller than the center coordinate of the proposal on the time axis, and the two do not overlap;

[0014] 3) The center coordinates of the neighbor's proposal are larger than the center coordinates of the proposal on the time axis, and the two overlap;

[0015] 4) The center coordinates of the neighbor proposal are greater than the center coordinates of the proposal on the time axis, and the two do not overlap;

[0016] 5) The neighbor proposal includes this proposal;

[0017] The above five types of neighbor proposals are used to construct proposal graphs at different granularities. The three construction methods are:

[0018] The first method is to select N neighbor proposals from the five types of neighbor proposals for each proposal, for a total of 5N neighbor proposals, and establish edges between them and the original proposal to construct a proposal graph;

[0019] Second, the five types of neighbor proposals are reclassified into three categories: the neighbor proposal in 5) is a separate category, the neighbor proposals in 1) and 2) belong to a category with a center coordinate smaller than the proposal, and the neighbor proposals in 3) and 4) both belong to a category with a center coordinate larger than the proposal. Then, N neighbor proposals are selected from each category of each proposal, for a total of 3N neighbor proposals, and edges are established between them and the original proposal to construct the proposal graph;

[0020] The third method is to re-divide these five types of neighbor proposals into two categories according to their center coordinates: neighbor proposals with center coordinates smaller than the proposal belong to one category, and other neighbor proposals belong to one category. Then, N neighbor proposals are selected from each category for each proposal, for a total of 2N neighbor proposals, and edges are established between them and the original proposal to construct a proposal graph.

[0021] In the above proposal graph construction method, the method of selecting N neighbor proposals is: using proposal p n Extension proposal The tIoU value is used to select the valuable N neighbor proposals in each category. n =(c p，n , l p，n ) The expression is as follows:

[0022]

[0023] where β is the expansion coefficient; the expression of tIoU is as follows:

[0024]

[0025] where I(p i , p j ) and U(p i , p j ) represent the time intersection and union between two proposals, respectively.

[0026] Step 2: Constructing a visual proposal graph network;

[0027] Using the proposal graph obtained in step 1, a L-layer visual proposal graph network is constructed using graph attention, and the attention weight is calculated according to the proposal feature similarity. To further mine the visual correlation between proposals, a multi-head attention mechanism is introduced in the visual graph convolution. The construction steps of multi-head visual graph convolution are as follows:

[0028] First, each proposal feature is divided into K parts along the channel dimension, and then the information of each part is connected along the channel dimension. The multi-head visual graph convolution is represented as:

[0029]

[0030] where represents the feature of the vth proposal on the lth visual convolution layer; represents the kth message feature vector obtained from the neighbor proposal of the vth proposal on the lth visual convolution layer; W l is a parameter matrix to be learned; || is a splicing operation; b visual represents the bias vector in the visual proposal graph network; N(v) represents the neighbor proposal of the vth proposal; represents the kth feature of the jth neighbor proposal on the l-1th visual convolution layer; is the kth visual head attention weight of the jth neighbor proposal of the proposal v, which is the cosine similarity between the visual feature of the vth proposal and the visual feature of the jth neighbor proposal, and is normalized. The calculation formula is as follows:

[0031]

[0032] where |N(v)| represents the number of neighbor proposals of the vth proposal; represents the transpose of the kth visual feature of the vth proposal, represents the kth visual feature of the jth neighbor proposal, and |||| represents the norm operation.

[0033] Step 3: Constructing the temporal proposal graph network

[0034] Using the proposal graph obtained in step 1, a temporal proposal graph network parallel to the visual proposal graph network described in step 2 is constructed, and a multi-head attention mechanism is introduced in the temporal graph convolution to extract information in different ways using different learning parameters for different categories of neighbor proposals. The construction steps of multi-head temporal graph convolution are as follows:

[0035] First, divide each proposal feature into K parts along the channel dimension, and then connect the information of each part along the channel dimension. The multi-head temporal graph convolution is represented as:

[0036]

[0037] where, represents the feature of the vth proposal on the lth temporal convolution layer; represents the kth message feature vector obtained from neighbor proposals for the vth proposal on the lth temporal convolution layer; and represent different parameter matrices to be learned; M is the number of neighbor proposal categories; N m (v) represents the mth neighbor proposal category of the vth proposal; b temporal represents the bias vector in the temporal proposal graph network; represents the kth feature of the jth neighbor proposal on the l-1th temporal convolution layer; is the kth temporal head attention weight of the jth neighbor proposal of proposal v, and the attention weight represents the importance of the neighbor proposal. To calculate the attention weight, a temporal position vector is constructed:

[0038] s vj = [t v，s , t v，e , l p，v , t j，s , t j，e , l p，j ] (7)

[0039] where s vj is the temporal position vector of the vth proposal and the jth neighbor proposal; t v，s and t v，e are the starting coordinates and ending coordinates of the vth proposal, respectively; t j，s and t j，e are the starting coordinates and ending coordinates of the jth neighbor proposal, respectively; l p，v is the length of the vth proposal, and l p，j is the length of the jth neighbor proposal.

[0040] The kth temporal head attention weight of the vth proposal and the jth neighbor proposal is calculated using the temporal position vector, and the calculation process is as follows:

[0041]

[0042] where W k is the parameter matrix to be learned; represents the kth temporal feature of the vth proposal; represents the kth temporal feature of the jth neighbor proposal; N b (v) represents the bth neighbor proposal class of the vth proposal; the function δ outputs a single vector representing the feature and can have various forms, such as addition, subtraction, concatenation, or Hadamard convolution.

[0043] In addition, in order to obtain valuable information from neighbor proposals, an auxiliary loss L a is designed as follows:

[0044]

[0045] where η is a hyperparameter; L is the number of temporal convolution layers; λ l is the weight coefficient of the fth temporal convolution layer; B is the batch size; represents the feature of the most recent real action of the vth proposal; is the parameter matrix to be learned; y v is the binary label of the vth proposal, when the tIoU between the vth proposal and its closest ground truth instance is greater than 0.7, y v is set to 1, if the tIoU is lower than 0.01, y v is set to 0; represents the message feature vector obtained from neighbor proposals by the vth proposal on the lth temporal convolution layer, and is represented as:

[0046]

[0047] where, is the parameter matrix to be learned; represents the feature of the jth neighbor proposal of the vth proposal on the l-1th temporal convolution layer.

[0048] Step 4: Constructing a temporal-visual proposal graph network (TVPGN);

[0049] The time-vision proposal graph network is composed of a base module, a proposal feature construction module and a proposal feature enhancement module. The input is a video feature sequence extracted by a pre-trained model and some proposals generated by a time sequence action proposal method. The output is the specific category label of the predicted action proposal after passing through the three modules in turn. The functions and specific operations of the three modules in the time-vision proposal graph network are as follows:

[0050] 1) The base module captures long-term past and future information, and improves the quality of frame features by using the bidirectional relationship between frame features. The base module uses a residual convolution block with two layers of bidirectional LSTM as the basic module to process the video feature sequence, and the given input feature sequence F ∈ R T×C The base module first extracts semantic information in the video frame using the residual convolution module, and then inputs the output of the residual convolution module into two layers of bidirectional LSTM, and connects the hidden state sequences from the forward and backward LSTM along the channel dimension as the final feature sequence.

[0051] 2) The proposal feature construction module uses a self-attention mechanism to model the proposal feature, and effectively maps various continuous time sequence action proposals to the same feature space. First, initialize a training vector cls, and calculate the pair-wise relationship between cls and all frame features in the proposal to obtain attention weights; then directly weight and combine all frame features with attention weights to obtain the proposal feature, and the training vector cls also contains some trained action information, which is added to the proposal feature as the final proposal feature;

[0052] 3) The proposal feature enhancement module receives the proposal feature generated by the proposal feature construction module, uses the TVPG module (composed of the time sequence proposal graph network obtained in step 2 and the vision proposal graph network obtained in step 3 in parallel) to obtain additional information between proposals to improve the quality of the proposal, and then uses an action classifier to predict the label of the proposal according to the enhanced proposal feature, and adds a completeness classifier to filter those proposals containing incomplete action instances. Specifically, a fully connected (FC) layer with a softmax operation is used as the action classifier, and n binary classifiers (each for an action class) are used as the completeness classifier. In order to avoid interference between the two classifiers during the training process, two TVPG modules (TVPG1 and TVPG2) are used in parallel to enhance the proposal feature, and the outputs of these modules are input into the two classifiers respectively.

[0053] Advantages

[0054] The application proposes a time-vision proposal graph network time action detection method, which uses a time graph convolution network and a vision graph convolution network on a proposal graph in parallel: first, select valuable neighbor proposals for each proposal, and construct them into an action proposal graph, then use a graph convolution network to obtain action information from neighbor proposals according to the different contributions of neighbor proposals to classification to construct a vision proposal graph network, and use the cosine similarity of the proposal features as the weight; at the same time, in order to utilize the time position relationship between the proposals, a time proposal graph network is constructed on the action proposal graph in parallel with the vision proposal graph network, and the information is aggregated through the time and vision correlation between the proposals. When using the graph convolution network, a new time graph convolution operation is proposed, which divides the neighbor proposals into multiple sub-neighbor according to the time position relationship, and extracts information in different ways, uses a measurement function as auxiliary supervision to obtain more valuable information from the neighbor proposals. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 For different time position relationship neighbor proposal graph types;

[0056] Figure 2 For time-vision proposal graph convolution operation schematic diagram;

[0057] Figure 3 For the overall architecture diagram of the time-vision proposal graph network;

[0058] Figure 4 For the visual time action detection result diagram on the THUMOS'14 dataset;

[0059] Figure 5 For the time action detection performance (mAP%) under different tIoU thresholds on THUMOS'14;

[0060] Figure 6 For the time action detection performance (mAP%) under different tIoU thresholds on ActivityNet-1.3. DETAILED DESCRIPTION

[0061] The application will be further described below in combination with the drawings and implementation cases.

[0062] The application proposes a time-vision proposal graph (TVPG) module and a time-vision proposal graph network (TVPGN) aiming at the fact that the existing algorithm usually only contains part of the ground truth action when generating proposals, and lacks enough semantic information to accurately predict their categories, which will be described in detail below.

[0063] 1. Time-vision proposal graph (TVPG) module.

[0064] 1.1 Constructing a proposal graph

[0065] The uncropped video V contains a set of action instances Φ g for:

[0066]

[0067] Among them, N g is the number of real action instances in video V, c gt，n , l gt，n They are action instances φ n The center coordinates and length, y gt，n For action instance φ n 's label;

[0068] The set of proposals Ψ generated in the untrimmed video V p for:

[0069]

[0070] Among them, N p is the number of proposals in video V, c p，n , l p，n The generated proposals p are n The center coordinates and length of

[0071] For Ψ p Each proposal p n , first of all, p In addition to the proposal n The other proposals are divided into different categories. The classification method is as follows: Figure 1 As shown in Figure 2, each proposal has five neighbor proposals based on the temporal position relationship:

[0072] 1) The center coordinate of the neighbor proposal is smaller than the center coordinate of the proposal on the time axis, and the two overlap;

[0073] 2) The center coordinate of the neighbor proposal is smaller than the center coordinate of the proposal on the time axis, and the two do not overlap;

[0074] 3) The center coordinates of the neighbor's proposal are larger than the center coordinates of the proposal on the time axis, and the two overlap;

[0075] 4) The center coordinates of the neighbor proposal are greater than the center coordinates of the proposal on the time axis, and the two do not overlap;

[0076] 5) The neighbor proposal includes this proposal;

[0077] The above five types of neighbor proposals are used to construct proposal graphs at different granularities. The three construction methods are:

[0078] The first method is to select N neighbor proposals from each of the five types of neighbor proposals, a total of 5N neighbor proposals, and build a proposal graph by establishing edges between them and the original proposal.

[0079] The second method is to reclassify the five types of neighbor proposals into three categories: 5) the neighbor proposals in the middle are a separate category, 1) and 2) the neighbor proposals with smaller center coordinates belong to one category, and 3) and 4) the neighbor proposals with larger center coordinates belong to another category. Then, for each proposal, select N neighbor proposals from each category, a total of 3N neighbor proposals, and build a proposal graph by establishing edges between them and the original proposal.

[0080] The third method is to reclassify the five types of neighbor proposals into two categories according to their center coordinates: the neighbor proposals with smaller center coordinates belong to one category, and the other neighbor proposals belong to another category. Then, for each proposal, select N neighbor proposals from each category, a total of 2N neighbor proposals, and build a proposal graph by establishing edges between them and the original proposal.

[0081] The above method of selecting N neighbor proposals is as follows:

[0082] For proposal p n =(c p，n , l p，n ), the extended proposal is as follows:

[0083]

[0084] where β is the expansion coefficient.

[0085] Assumption: The closer two proposals are on the time axis, the more likely they come from the same action instance. Based on this assumption, the tIoU of the extended proposal is used to select valuable neighbor proposals in each category, which is expressed as follows:

[0086]

[0087] where I(p i , p j ) and U(p i , p j ) represent the time intersection and union of the two proposals, respectively.

[0088] 1.2 Building a visual proposal graph network

[0089] The L-layer visual proposal graph network is constructed by using graph attention, and the attention weight is calculated according to the similarity of the proposal features. In order to further mine the visual correlation between proposals, a multi-head attention mechanism is introduced in the graph convolution. First, each proposal feature is divided into K parts along the channel dimension, and then the information of each part is connected along the channel dimension. The multi-head visual graph convolution is represented as:

[0090]

[0091] wherein, represents the feature of the vth proposal on the lth visual convolution layer; represents the kth message feature vector obtained from the neighbor proposal on the lth visual convolution layer of the vth proposal; W l is a parameter matrix to be learned; || is a splicing operation; b visual represents a bias vector in the visual proposal graph network; N(v) represents the neighbor proposal of the vth proposal; represents the kth feature of the jth neighbor proposal on the l-1th visual convolution layer; is the kth visual head attention weight of the jth neighbor proposal of the proposal v, which is the cosine similarity between the visual feature of the vth proposal and the visual feature of the jth neighbor proposal, and is normalized. The calculation formula is as follows:

[0092]

[0093] wherein, |N(v)| represents the number of neighbor proposals of the vth proposal; represents the transpose of the kth visual feature of the vth proposal, represents the kth visual feature of the jth neighbor proposal, and ||| represents the norm operation.

[0094] 1.3 Construction of the time sequence proposal graph network

[0095] The time sequence position relationship plays an important role in information transmission between proposals. Since the number of node connections is large and the nodes are unordered, the graph convolution cannot utilize the position relationship between the nodes. Therefore, a new attention-based time sequence graph convolution is designed, which uses different learning parameters for different categories of neighbor proposals and extracts information in different ways.

[0096] A multi-head attention mechanism is introduced in the time sequence graph convolution to construct a time sequence proposal graph network in parallel with the visual proposal graph network, as shown in Figure 2 . First, each proposal feature is divided into K parts along the channel dimension, and then the information of each part is connected along the channel dimension. The multi-head time sequence graph convolution is represented as:

[0097]

[0098] where, denotes the feature of the vth proposal on the lth temporal convolutional layer; denotes the kth message feature vector of the vth proposal on the lth temporal convolutional layer from neighbor proposals; and is the parameter matrix to be learned; M is the number of neighbor proposal classes; N m (v) denotes the bth neighbor proposal class of the vth proposal; b temporal denotes the bias vector in the temporal proposal graph network; denotes the kth feature of the jth neighbor proposal on the l-1th temporal convolutional layer; is the kth temporal head attention weight of the jth neighbor proposal of the proposal v, which indicates the importance of the neighbor proposal. To calculate the attention weight, a temporal position vector is constructed:

[0099] s vj = [t v，s , t v，e , l p，v , t j，s , t j，e , l p，j ] (8)

[0100] where s vj is the temporal position vector of the vth proposal and the jth neighbor proposal; t v，s and t v，e are the start coordinate and end coordinate of the vth proposal, respectively; t j，s and t j，e are the start coordinate and end coordinate of the jth neighbor proposal, respectively; l p，v is the length of the vth proposal, and l p，j is the length of the jth neighbor proposal.

[0101] The kth temporal head attention weight of the vth proposal and the jth neighbor proposal is calculated using the temporal position vector:

[0102]

[0103] where W k is the parameter matrix to be learned; denotes the kth temporal feature of the vth proposal; denotes the kth temporal feature of the jth neighbor proposal; N b (v) denotes the bth neighbor proposal class of the vth proposal; the function δ outputs a single vector indicating the feature and , which can have various forms such as addition, subtraction, concatenation, or Hadamard convolution.

[0104] The smaller the distance between the information feature of the real action instance and the proposal feature, the more valuable the obtained information is. In order to obtain valuable information from neighbor proposals, a contrastive loss is adopted to design the auxiliary loss. The auxiliary loss L a is designed as:

[0105]

[0106] where η is a hyper-parameter; L is the number of temporal convolutional layers; λ l is the weight coefficient of the f-th temporal convolutional layer; B is the batch size; represents the feature of the nearest real action of the v-th proposal; is the parameter matrix to be learned; y v is the binary label of the v-th proposal, when the tIoU between the v-th proposal and its closest ground truth instance is greater than 0.7, y v is set to 1, if the tIoU is lower than 0.01, y v is set to 0; represents the message feature vector obtained from neighbor proposals by the v-th proposal on the l-th temporal convolutional layer, which is represented as:

[0107]

[0108] where, is the parameter matrix to be learned; represents the feature of the j-th neighbor proposal of the v-th proposal on the l-1-th temporal convolutional layer.

[0109] 2. Temporal-visual proposal graph network (TVPGN)

[0110] The temporal-visual proposal graph network is composed of three modules, namely, a base module, a proposal feature construction module and a proposal feature enhancement module, as shown in Figure 3 The input is a video feature sequence extracted by a pre-trained model and some proposals generated by a temporal action proposal method, which sequentially passes through the three modules, and the output is the specific class label of the predicted action proposal.

[0111] 1) The base module captures long-term past and future information, and improves the quality of frame features by using the bidirectional relationship between frame features. This module uses a residual convolutional block with two layers of bidirectional LSTM as the base module to process the video feature sequence, given the input feature sequence F ∈ R T×CThe base module first extracts semantic information in the video frame using a residual convolution module, and then inputs the output of the residual convolution module into two layers of bidirectional LSTM, and connects the hidden state sequences from the forward and backward LSTM along the channel dimension as the final feature sequence.

[0112] 2) The proposal feature construction module uses a self-attention mechanism to model the proposal feature, and effectively maps various continuous time action proposals to the same feature space. First, initialize a training vector cls, and calculate the pair relationship between cls and all frame features in the proposal to obtain the attention weight; then directly weight and combine all frame features with the attention weight to obtain the proposal feature, and the training vector cls also contains some trained action information, which is added to the proposal feature as the final proposal feature;

[0113] 3) The proposal feature enhancement module receives the proposal feature generated by the proposal feature construction module, uses the TVPG module (composed of a parallel time proposal graph network and a visual proposal graph network) to obtain additional information between proposals to improve the quality of the proposal, and then uses an action classifier to predict the label of the proposal according to the enhanced proposal feature, and adds a completeness classifier to filter those proposals containing incomplete action instances. Specifically, a fully connected (FC) layer with a softmax operation is used as the action classifier, and n binary classifiers (each for an action class) are used as the completeness classifier. In order to avoid interference between the two classifiers during the training process, two TVPG modules (TVPG1 and TVPG2) are used in parallel to enhance the proposal feature, and the outputs of these modules are input into the two classifiers respectively.

[0114] The experimental verification results are introduced below, and the method of generating the training data set and the test data set is as follows:

[0115] Primary proposal: In order to generate training data, first execute the time action proposal generation model on the videos in the training set to obtain a large number of candidate proposals. Then remove the proposals that do not overlap with the ground truth instances as training data. For inference data, use the time action proposal generation model to generate proposals with confidence scores, and then use the soft net method to change their confidence scores. According to the confidence score, select the top D proposals from each video as the final proposal.

[0116] Label definition: After getting proposals, they are classified into three types by calculating their tIoU with ground truth action instances: (1) positive proposals with the largest tIoU greater than 0.7 with the ground truth action instance; (2) background proposals with the smallest tIoU less than 0.01 with the closest ground truth action instance; (3) incomplete proposals with the tIoU less than 0.3 with the instance. Positive and background proposals are used to train the action classifier; positive and incomplete proposals are applied to the integrity classifier.

[0117] Loss function: The network is trained using a multi-task loss function. The total loss function is represented as:

[0118]

[0119] Where λ1, λ2, λ3, λ4, λ5 are weight coefficients; L c is the cross-entropy loss function; L com is the hinge loss for each binary classifier of the integrity classifier; the regression loss L reg is designed as Smooth-L1 loss.

[0120] Figure 4 The detection results visualized on the THUMOS'14 dataset are shown. Examples are selected from test videos, each of which consists of some frame sequences. Comparing the proposed method with three baseline detection results, it can be seen that the detection of the present application has more accurate timing boundaries. Since TVPGN obtains information from neighbor proposals to expand the receptive field of each proposal, which may contain the entire ground truth action instance, it has enough information to generate accurate timing boundaries.

[0121] The effectiveness is verified on two datasets, THUMOS'14 and ActivityNet-1.3, and the comparison results of the proposed TVPGN method with other state-of-the-art temporal action detection methods are shown in Figure 5 and Figure 6 .

[0122] Figure 5The detection performance comparison of TVPGN and the temporal action detection algorithms BSN, AGCN, BMN, G-TAD, POLP, MDN, P-GCN, SALAD, I2Net, AGT, PCG-TAL, VSGN, DaoTAD, AFSD, SP-TAD, MUSES, AVFusion and TAPP on the THUMOS'14 dataset is given, and the proposals generated by different temporal action proposal generation methods are used as the input of temporal action detection. It can be seen that when combined with BMN-TAPP, the performance of TVPGN is significantly better than the previous advanced methods. In particular, the mAP index of TVPGN reaches 65.2% at tIoU=0.5, which is improved by 1.0% (from 64.2% to 65.2%) compared with the previous best method (TAPP) on the THUMOS'14 dataset.

[0123] Figure 6 The temporal action detection method comparison results of various methods on the ActivityNet-1.3 dataset are given. When tIoU=0.95, the performance of TVPGN is better than the previous most advanced method no matter which method is used to generate proposals as input.

Claims

1. A method for temporal action detection of a temporal-visual proposal graph network, characterized in that The steps of the method include: Step 1: constructing a proposal graph in the uncropped video V; Step 2: constructing a visual proposal graph network according to the proposal graph obtained in step 1; Step 3: constructing a temporal proposal graph network according to the proposal graph obtained in step 1; Step 4: constructing a temporal-visual proposal graph network according to the visual proposal graph network constructed in step 2 and the temporal proposal graph network constructed in step 3, and performing temporal action detection using the constructed temporal-visual proposal graph network; In step 3, the temporal proposal graph network parallel to the visual proposal graph network is constructed using the proposal graph obtained in step 1, and the specific method is: First, divide each proposal feature into K parts along the channel dimension, and then connect the information of each part along the channel dimension, then the multi-head temporal graph convolution of the temporal proposal graph network is represented as: wherein, represents the feature of the vth proposal on the lth temporal convolutional layer; represents the kth message feature vector of the vth proposal on the lth temporal convolutional layer from neighbor proposals; and represents different parameter matrices to be learned; M is the number of neighbor proposal classes; N m (v) represents the mth neighbor proposal class of the vth proposal; b temporal represents the bias vector in the temporal proposal graph network; represents the kth feature of the jth neighbor proposal on the l-1th temporal convolutional layer; is the kth temporal head attention weight of the jth neighbor proposal of the proposal v, the attention weight represents the importance of the neighbor proposal, to calculate the attention weight, a temporal position vector is constructed: s vj = [t v,s , t v,e , l p,v , t j,s , t j,e , l p,j ](7) where s vj is the temporal position vector of the vth proposal and jth neighbor proposal; t v,s and t v,e are the start and end coordinates of the vth proposal, respectively; t j,s and t j,e are the start and end coordinates of the jth neighbor proposal, respectively; l p,v is the length of the vth proposal, and l p,j is the length of the jth neighbor proposal. The temporal position vector is used to calculate the kth temporal head attention weight of the vth proposal and the jth neighbor proposal, and the specific calculation process is as follows: where W k is the parameter matrix to be learned; denotes the kth temporal feature of the vth proposal; denotes the kth temporal feature of the jth neighbor proposal; N b (v) denotes the bth neighbor proposal class of the vth proposal; the function δ outputs a single vector representing the features and of the bth neighbor proposal class of the vth proposal. To obtain valuable information from the neighbor proposals, a design assistant loss L a is defined as: where η is a hyper-parameter; L is the number of temporal convolutional layers; λ l is the weight coefficient of the l-th temporal convolutional layer; B is the batch size; represents the feature of the v-th proposal’s nearest ground truth instance; is the parameter matrix to be learned; y v is the binary label of the v-th proposal, which is set to 1 when the tIoU between the v-th proposal and its nearest ground truth instance is greater than 0.7, and 0 otherwise; v is set to 1 if the tIoU is lower than 0.01, and 0 otherwise; v is set to 0; represents the message feature vector of the v-th proposal obtained from its neighbors at the l-th temporal convolutional layer, which is represented as: wherein, is a parameter matrix to be learned; denotes the feature of the j-th neighbor proposal of the v-th proposal on the l-1-th temporal convolutional layer.

2. The temporal action detection method of the temporal-visual proposal graph network according to claim 1, characterized in that: The specific method for constructing the proposal graph in the uncropped video V in step 1 is: constructing the proposal graph in the uncropped video V according to the generated proposal set Classifying to obtain a proposal set Ψ p Each proposal p n in the proposal set Ψ p has five types of neighbor proposals, and the proposal graph is constructed according to the obtained five types of neighbor proposals; in the formula, N p,n is the number of generated proposals in the video V, c p,n and l n are the center coordinates and length of the generated proposal p n , respectively.

3. The temporal action detection method of the temporal-visual proposal graph network according to claim 2, characterized in that: The set of proposals Ψ p The method of sorting is: according to the time sequence position relationship, each proposal p n There are five kinds of neighbor proposals: 1) The center coordinates of the neighbor proposal are smaller than the center coordinates of the proposal on the time axis, and the two overlap; 2) The center coordinates of the neighbor proposal are smaller than the center coordinates of the proposal on the time axis, and the two do not overlap; 3) The center coordinates of the neighbor proposal are greater than the center coordinates of the proposal on the time axis, and the two overlap; 4) The center coordinates of the neighbor proposal are greater than the center coordinates of the proposal on the time axis, and the two do not overlap; 5) The neighbor proposal contains the proposal.

4. The temporal action detection method of the temporal-visual proposal graph network according to claim 3, characterized in that: The method for constructing the proposal graph according to the five types of neighbor proposals obtained is: For each proposal, select N neighbor proposals from each of the five types of neighbor proposals, a total of 5N neighbor proposals, and establish edges between them and the original proposal to construct the proposal graph.

5. The temporal action detection method of the temporal-visual proposal graph network according to claim 3, characterized in that: The method for constructing the proposal graph according to the five types of neighbor proposals obtained is: The five types of neighbor proposals are re-divided into three categories: 5) in the neighbor proposal is a separate category, 1) and 2) in the neighbor proposal belong to the category with smaller center coordinates than the proposal, and 3) and 4) in the neighbor proposal belong to the category with larger center coordinates than the proposal, then select N neighbor proposals from each category of each proposal, a total of 3N neighbor proposals, and establish edges between them and the original proposal to construct the proposal graph.

6. The temporal action detection method of the temporal-visual proposal graph network according to claim 3, characterized in that: The method for constructing the proposal graph according to the five types of neighbor proposals obtained is: The five types of neighbor proposals are re-divided into two categories according to their center coordinates: the neighbor proposals with center coordinates less than the proposal belong to one category, and the other neighbor proposals belong to another category, then N neighbor proposals are selected for each proposal from each category respectively, a total of 2N neighbor proposals, and edges are established between them and the original proposal to construct the proposal graph.

7. The method of claim 4, 5 or 6, wherein the method further comprises: The method of selecting N neighbor proposals is to select N neighbor proposals with the highest tIoU values of the extended proposals of the proposals p n for each class, where the extended proposal n = (c p,n , l p,n ) of the proposal p i is expressed as follows:​ wherein β is an expansion coefficient; and tIoU is expressed as follows: where I(p i ,p j ) and U(p i ,p j ) denote the intersection and union of the two proposals, respectively.

8. The method of claim 1, wherein the method further comprises: The specific steps of constructing the visual proposal graph network in step 2 using the proposal graph obtained in step 1 are as follows: First, each proposal feature is divided into K parts along the channel dimension, and then the information of each part is connected along the channel dimension, and the multi-head visual graph convolution of the visual proposal graph network is expressed as: where, represents the feature of the vth proposal on the lth visual convolution layer; represents the kth message feature vector obtained from neighbor proposals of the vth proposal on the lth visual convolution layer; W l is the parameter matrix to be learned; || is the splicing operation; b visual represents the bias vector in the visual proposal graph network; N(v) represents the neighbor proposals of the vth proposal; represents the kth feature of the jth neighbor proposal on the l-1th visual convolution layer; is the kth visual head attention weight of the jth neighbor proposal of the proposal v, which is the cosine similarity of the vth proposal visual feature and the jth neighbor proposal visual feature normalized, and the calculation formula is as follows: where |N(v)| represents the number of neighbor proposals of the vth proposal; represents the transpose of the kth visual feature of the vth proposal, represents the kth visual feature of the jth neighbor proposal, and || || represents the norm operation.

9. The method of claim 1, wherein the method further comprises: The time-vision proposal graph network constructed in step 4 is composed of three modules: a base module, a proposal feature construction module, and a proposal feature enhancement module. The input is the video feature sequence extracted by the pre-trained model and the proposals generated by some temporal action proposal method. The output is the specific category label of the predicted action proposal after passing through the three modules in turn. The base module is used for capturing long-time past and future information, improving frame feature quality by using bidirectional relationship between frame features, and adopting a residual convolution block with two layers of bidirectional LSTM as a base module to process a video feature sequence, given an input feature sequence F T×C The base module first extracts semantic information in a video frame using a residual convolution module, then inputs the output of the residual convolution module into two layers of bidirectional LSTM, and connects hidden state sequences from the forward and backward LSTM along a channel dimension as a final feature sequence. The proposal feature construction module uses a self-attention mechanism to model the proposal feature, effectively mapping various continuous temporal action proposals to the same feature space. First, a training vector cls is initialized, and the pairwise relationship between cls and all frame features in the proposal is calculated to obtain attention weights. Then, all frame features are directly weighted and summed with the attention weights to obtain the proposal feature. The training vector cls also contains some trained action information, which is added to the proposal feature as the final proposal feature. The proposal feature enhancement module receives the proposal feature generated by the proposal feature construction module, and in parallel uses the temporal proposal graph network obtained in step 2 and the visual proposal graph network obtained in step 3 to obtain additional information between the proposals. Then, an action classifier is used to predict the label of the proposal based on the enhanced proposal feature, an integrity classifier is added to filter proposals containing incomplete action instances. A fully connected FC layer with a softmax operation is used as the action classifier, and n binary classifiers are used as the integrity classifier. In parallel, the visual proposal graph network and the temporal proposal graph network are used to enhance the proposal feature, and the outputs of these modules are input into the two classifiers respectively.

Images