Feature migration-based category incremental learning weak supervision time sequence action positioning method
Through the category incremental learning method based on feature transfer, the problem of low accuracy of the timing action positioning method under weak supervision and insufficient model expansion ability is solved, the precise positioning and classification of video actions is realized, and the learning ability of the model is improved. Positioning accuracy.
Patent Information
- Application Number
- CN202411064110.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-06
AI Technical Summary
The existing timing action positioning methods have low positioning accuracy in action instances in untrimmed videos and cannot realize the extended learning ability of the model. Especially under weak supervision, the lack of frame-level tags seriously affects the positioning accuracy.
Using the class incremental learning method based on feature transfer, the video input module, task ID generation module, adjacent task classifier learning module and positioning and post-processing module based on feature transfer can be realized accurately positioning and classification of video actions, and using The feature transfer mechanism realizes data distribution alignment in the feature domain, avoids catastrophic forgetting of class information, and enhances the extended learning ability of the model.
It improves the accuracy of timing action positioning and the extended learning ability of the model, reduces the difficulty of task general classifiers to accurately locate multi-category actions, and improves classification accuracy and ability to adapt to data changes.
Smart Images

Figure CN119091497A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of deep learning and relates to a method for weakly supervised temporal action localization based on category incremental learning of feature migration. Background Art
[0002] With the rapid development of monitoring equipment and Internet short video platforms, the number of videos generated has exploded. The temporal action localization method based on video analysis is widely used in intelligent monitoring, intelligent driving, and interesting video recommendation.
[0003] Existing temporal action localization methods mainly include two methods: full supervision and weak supervision. The full supervision method has the temporal position label and category label of the action as supervision during the training process, but it is difficult to obtain the actual position label in reality, so it has great limitations in practical applications. Weakly supervised positioning learning is a method of positioning the action time period (defined as a set of continuous time points related to the action event) using only sequence-level coarse-grained annotations (weak labels). This method is more in line with practical applications, so the present invention is a weakly supervised temporal action localization method. However, the lack of frame-level labels seriously affects the positioning accuracy of the action instances of untrimmed videos, and the existing methods cannot allow the model to have the ability to expand learning, so the study of efficient and accurate weakly supervised temporal action localization methods has important scientific research and engineering application significance. Summary of the invention
[0004] The purpose of the present invention is to provide a method for weakly supervised temporal action localization based on feature migration and incremental learning, which can improve the accuracy of video localization and action recognition.
[0005] The technical solution adopted by the present invention is a method for weakly supervised temporal action localization based on feature migration and incremental learning, which specifically includes the following steps:
[0006] Step 1, obtain a video data set and a corresponding true value label set, and preprocess the video data set, divide the preprocessed video data set into a training video set and a test video set, and divide the corresponding true labels after the preprocessing of the video data set into a training set true value label and a test set true value label;
[0007] Step 2: Construct a category incremental learning weakly supervised temporal action localization network structure FTIL based on feature transfer;
[0008] Step 3, design the network loss function Los;
[0009] Step 4, training the network model M;
[0010] Step 5: Test model M on the test video set to obtain the action localization results of the video and evaluate the localization results.
[0011] The present invention is also characterized in that:
[0012] The specific process of step 1 is:
[0013] Download the action video set VDS from the official website of temporal action detection. 1 ,…,V m ,…,V M} and the corresponding action truth label set LS = {Y 1 ,…,Y m ,…,Y M},V m and Y m represents the mth video in VDS and its corresponding action category label, 1≤m≤M, M represents the total number of videos in VDS, W, H, T m Corresponding to V m The width, height and video duration of the image frame. Represents Y m The nth m action instance labels, 1≤n m ≤N m , N m Represents Y m The total number of action instances in ;
[0014] In the training set, the true value label L Tra middle in, Takes 0 or 1, representing the cth video in the mth video. m Whether the class action exists, C m Indicates the total number of categories;
[0015] In the test set, the true value label L Tes middle in Respectively The start time, end time and action category number;
[0016] Divide VDS into training videos T according to the ratio of m:n ra S and test video T es S, Indicates T ra The pth video in S, Indicates T es The qth video in S, P+Q=M, where m>m;
[0017] Press LS to T ra S and Tes S is divided into the training set truth label L Tra and the test set true value label L Tes , Indicates L Tra The p-th video truth label in , Indicates L Tes The qth video truth label.
[0018] In step 2, the feature transfer-based category incremental learning weak supervision network structure for temporal action positioning includes a video input module, a task ID generation module, a feature transfer-based adjacent task classifier learning module, and a positioning and post-processing module.
[0019] The specific process of step 2 is:
[0020] Step 2.1, Video Input Module: From T ra Extract the pth video from S All clips of and its corresponding label As network input, 1 <p<P;
[0021] Step 2.2, Task ID Generation Module: and its corresponding label As input to the task ID generation module, we get the training video clips The task ID number n;
[0022] Step 2.3, adjacent task classifier learning module based on feature transfer: according to the task number n, As input to the nth feature extractor FE n and the n-1th feature extractor FE n-1 In the example, we select the first to ninth layers of I3D as the backbone network for feature extraction, and use the output of the ninth layer as The deep features F, T, D represent the video length and depth feature dimension respectively, and the depth feature F is fed into the nth classifier Cla n , the classifier consists of a series of convolutional layers and a sigmoid layer. n Output classification results represents the probability score that the action belongs to the i-th category, 1 <i<c n , c n Represents the nth classifier Cla n The total number of predictable action categories;
[0023] Step 2.4, Positioning and post-processing module: The video is sent as input to the positioning and post-processing module, and the video clips with scores higher than the preset threshold are selected, and the continuous clips are merged into a positioning result. and denote the start time and end time of the predicted action instance in the pth video, respectively. represents the predicted action category of the i-th action instance in the p-th video.
[0024] In step 2.2, The process of generating a task ID number is as follows:
[0025] The total number of task ID numbers is set to N, the total number of category tag numbers is set to H, and the corresponding linear table L between category tag numbers and task ID numbers is manually established according to the tag information corresponding to the category tag number. Enter the action category tag number h, call the linear table query function LocatedElem(L,h), and you can query it in table L The task ID number n.
[0026] In step 2.3, the adjacent task classifier learning module based on feature transfer includes different learning according to different task ID numbers, and the specific process is as follows:
[0027] (1) Get the current fragment The task ID number n;
[0028] (2) Determine whether n is equal to 1. If so, put the fragment Then, they are fed into the first feature extractor FE. 1 , the first classifier Cla 1 , and get the output Indicates the probability score that the video action belongs to the xth category, 1 <x<c 1 , c 1 Represents the first classifier Cla 1 The total number of predictable action categories, proceed to step 2.4;
[0029] (3) Otherwise, put the fragment Feed into the nth and n-1th feature extractors FE n and FE n-1 , and get the corresponding features and The superscript n indicates the corresponding feature extractor number, and the subscript n corresponds to the task number. Feed it into the nth classifier Cla n , output represents the probability score that the action belongs to the i-th category, 1 <i<cn , c n Represents the nth classifier Cla n The total number of predictable action categories, go to step 2.4, and Send to the nth feature migration module Output post-migration features Bundle and are fed into the generator G output and Then and They are sent to the discriminator D respectively, and the output task source prediction result σ, σ = (ρ n ,ρ n-1 ), ρ n and ρ n-1 They represent the probability values of the feature source belonging to task n and task n-1 respectively.
[0030] In step 3, the total loss Los includes the classification loss L class , cosine similarity loss L cos and adversarial loss L adv , as follows:
[0031] Los=L class +L cos +L adv (1)
[0032]
[0033] in, and They represent the true value label and prediction result of the p-th video belonging to the c-th action in task n, respectively. n represents the total number of action categories under task n, mean() represents the averaging function, t is the true value label of the feature task source of the discriminator, and σ is the predicted value of the feature task source of the discriminator.
[0034] The specific process of step 4 is:
[0035] Step 4.1, set the network model training parameters, that is, set the learning rate variable l, the maximum number of training iterations variable Max_Epoch, the size of each batch of data variable Batch_Size, the minimum change value of the target loss function ΔMini_Loss, the number of training iterations variable step, and initialize step = 1;
[0036] Step 4.2: Input the training video sample set T ra S, true value label training set L Tra ;
[0037] Step 4.3, from the training video sample set T ra S and the true value label training set L Tra Randomly read Batch Size training samples and the true value label The read data is sent to the network constructed in step 2 for network training. In the training process of the discriminator D, the gradient reversal strategy is used, that is, the gradient reversal layer. The gradient reversal layer multiplies the back-propagated gradient by -λ. λ changes from 0 to 1 as the number of training generations increases. p represents the ratio of the current generation step to the total training generation Max_Epoch;
[0038] Step 4.4, calculate the absolute difference diff of the loss function Los of two iterations during the network training process, and judge whether the training model converges according to diff. If diff < ΔMini_Loss or step > Max_Epoch, the model converges, otherwise the model does not converge;
[0039] Step 4.5: If the model converges, go to step 4.6; if the model does not converge, step = step + 1, use the Adam optimizer to reversely correct the weight coefficients of each network layer in the network model, continue training, and return to step 4.3;
[0040] Step 4.6, network training is completed and the training model M of the end-to-end network is saved.
[0041] The specific process of step 5 is:
[0042] Step 5.1: Input the test video sample set T es S, true value label test set L Tes And the trained network model M;
[0043] Step 5.2: Define the time sequence action positioning result set variable R e , sequential action positioning result variable R er , test video sample set T es S's total sample number variable R, test sample number counter variable r, R e Initialized to empty, that is, R e = NULL, r is initialized to 1, r = 1;
[0044] Step 5.3: Take the test video sample set T es The rth video in S is sent to M to output the temporal action localization result set R er ;
[0045] Step 5.4, R er Append to R e Middle, that is, Re + = R er ;
[0046] Step 5.5, determine whether the number of r in the current test video is greater than or equal to R. If so, the test ends and outputs R e , otherwise r=r+1, return to step 5.3;
[0047] Step 5.6: The positioning results of all test videos are evaluated by calculating the mean average precision mAP to obtain the evaluation result Pa. The calculation formula is shown in (5):
[0048] Pa=mAP(R e ,L Tes ) (5)
[0049] Among them, mAP(*,*) is the mean average precision calculation function, R e and L Tes Represent the test video sample set T es S’s temporal action localization result set and the corresponding true value label test set.
[0050] The specific process of step 5.3 is:
[0051] (1) Define the task number variable nn and initialize nn = N;
[0052] (2) Take the test video sample set T es The rth video in S is fed into the Nth feature extractor FE N , get the video features
[0053] (3) Video features Feed into the Nth classifier Cla N , and get the classification results represents the probability score that the action belongs to the i-th category, 1 <i<c N , c N Represents the Nth classifier Cla N The total number of predictable action categories;
[0054] (4) Find the maximum value and the corresponding category number. max() represents the maximum value function;
[0055] (5) If Enter the positioning and post-processing module, select video clips with scores higher than the preset threshold, merge continuous clips into a positioning result and output the positioning result, exit the test, otherwise enter the Nth feature migration module Get the post-migration features thre is the preset threshold;
[0056] (6) nn = nn-1, the migrated features As input, go to step (3).
[0057] The beneficial effect of the present invention is to improve the performance of temporal action positioning. The present invention proposes a method for weakly supervised temporal action positioning based on feature migration and incremental learning. The action categories are divided into different tasks according to the different fields (sports, life, etc.) to which the action belongs. In order to expand the model as the number of tasks increases, the classifier under a specific task is incrementally trained to enable it to have sufficient ability to adapt to data changes. When learning new tasks, the feature migration mechanism is used to achieve data distribution alignment between different tasks in the feature domain to avoid catastrophic forgetting of class information. This method can reduce the problem that the general task classifier cannot accurately locate all categories of actions, improve the classification accuracy, and enable the model to have category expansion learning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 It is a flow chart of the method for weakly supervised temporal action localization based on feature migration and category incremental learning of the present invention;
[0059] Figure 2 It is a flow chart of the network for weakly supervised temporal action localization based on feature migration and category incremental learning of the present invention;
[0060] Figure 3 It is a flow chart of a migration module of adjacent tasks in an adjacent task classifier learning network based on feature migration of the present invention;
[0061] Figure 4 It is a test flow chart of the network for weakly supervised temporal action localization based on feature migration and category incremental learning of the present invention;
[0062] Figure 5-1 to Figure 5-3 This is a subjective effect diagram of applying the class incremental learning weakly supervised temporal action localization method based on feature migration of the present invention to the THUMOS14 dataset. Figure 5-1 is the positioning result of the harmonica action instance, Figure 5-2 is the positioning result of the javelin throwing action instance, Figure 5-3 is the localization result of the instance of cricket bowling action;
[0063] Figure 6-1~ Figure 6~3 is a subjective effect diagram of applying the weakly supervised temporal action localization method of the present invention based on feature migration based on category incremental learning to the ActivityNet1.3 dataset, Figure 6-1 is the positioning result of the Maverick lasso action instance, Figure 6-2 is the positioning result of the carpet installation action instance, Figure 6-3 It is the localization result of the beach soccer action instance. DETAILED DESCRIPTION
[0064] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0065] The present invention is based on the feature migration category incremental learning weak supervision time sequence action localization method, the overall process is as follows Figure 1 As shown, the specific implementation steps are as follows:
[0066] Step 1: Download the action video set VDS from the official website of time-series action detection. 1 ,…,V m ,…,V M} and the corresponding action truth label set LS = {Y 1 ,…,Y m ,…,Y M},V m and Y m represents the mth video in VDS and its corresponding action category label, 1≤m≤M, M represents the total number of videos in VDS, M=1000, W, H, T m Corresponding to V m The width, height and video duration of the image frame W = H = 112, T m =512, Represents Y m The nth m action instance labels, 1≤n m ≤N m , N m Represents Y m The total number of action instances in the training set truth label L Tra middle in It can be 0 or 1, indicating whether the c-th action exists, C represents the total number of categories, C = 20, in the test set true value label L Tes middle in Respectively The start time, end time and action category of VDS are divided into training videos T according to the ratio of m:n (m>m). ra S and test video T esS, m = n = 500, Indicates T ra The pth video in S, Indicates T es The qth video in S, P+Q=M, P=Q=500, press T to turn LS ra S and T es S is divided into the training set truth label L Tra and the test set true value label L Tes , Indicates L Tra The p-th video truth label in , Indicates L Tes The qth video truth label.
[0067] Step 2, design the network structure of the weakly supervised temporal action localization based on feature migration and category incremental learning proposed by the present invention for temporal action localization, such as Figure 2 The network structure shown in FIG. 1 is composed of a video input module, a task ID generation module, a feature migration-based adjacent task classifier learning module, and a positioning and post-processing module. When the task ID is n, the adjacent task migration module flow chart of the adjacent task classifier learning network based on feature migration of the present invention is as follows: Figure 3 As shown; the specific process of step 2 is:
[0068] Step 2.1, video input module, from T ra Extract the pth video from S All clips of and its corresponding label As network input, 1 <p<P;
[0069] Step 2.2, task ID generation module, and its corresponding label As input to the task ID generation module, we get the training video clips The specific process is as follows: the total number of task ID numbers is set to N, N = 2, the total number of category label numbers is H, H = 20, and the corresponding linear table L between the category label number and the task ID number is manually established according to the label information corresponding to the category label number. The action category label number h is input, and the linear table query function LocatedElem (L, h) is called. The corresponding linear table L can be queried. The task ID number n is as follows: Table 1 shows the correspondence between some category labels and task ID numbers
[0070] Table 1
[0071]
[0072] Step 2.3, the adjacent task classifier learning module based on feature migration, according to the task number n As input to the nth feature extractor FE n and the n-1th feature extractor FE n-1 In the example, we select the first to ninth layers of I3D as the backbone network for feature extraction, and use the output of the ninth layer as The deep features F, Denote the video length and depth feature dimension respectively, where T = 512, D = 2048, and the depth feature F is fed into the nth classifier Cla n , the classifier consists of a series of convolutional layers and a sigmoid layer. n Output classification results represents the probability score that the action belongs to the i-th category, 1 <i<c n , c n Represents the nth classifier Cla n The total number of predictable action categories; the specific process of step 2.3 is:
[0073] (1) Get the current fragment The task ID number n;
[0074] (2) Determine whether n is equal to 1. If so, put the fragment Then, they are fed into the first feature extractor FE. 1 , the first classifier Cla 1 , and get the output Indicates the probability score that the video action belongs to the xth category, 1 <x<c 1 , c 1 Represents the first classifier Cla 1 The total number of predictable action categories, proceed to step 2.4;
[0075] (3) Otherwise, put the fragment Feed into the nth and n-1th feature extractors FE n and FE n-1 , and get the corresponding features and The superscript n indicates the corresponding feature extractor number, and the subscript n corresponds to the task number. Feed it into the nth classifier Cla n , output represents the probability score that the action belongs to the i-th category, 1 <i<c n , c nRepresents the nth classifier Cla n The total number of predictable action categories, go to step 2.4, and Send to the nth feature migration module Output post-migration features Bundle and They are sent to the adversarial network (composed of a generator G and a discriminator D). The function of the generator G is feature replication. The discriminator D consists of four cascaded fully connected layers. The function of the discriminator D is to determine the task source of the feature, that is: and are fed into the generator G output and Then and They are sent to the discriminator D respectively, and the output task source prediction result σ, σ = (ρ n ,ρ n-1 ), ρ n and ρ n-1 They represent the probability values of the feature source belonging to task n and task n-1 respectively; Table 2 shows the nth classifier parameters, Table 3 shows the nth feature transfer module parameters, and Table 4 shows the discriminator parameters:
[0076] Table 2
[0077]
[0078]
[0079] Table 3
[0080]
[0081] Table 4
[0082]
[0083] Step 2.4, positioning and post-processing module, The video is sent as input to the positioning and post-processing module, and the video clips with scores higher than the preset threshold are selected, and the continuous clips are merged into a positioning result. and denote the start time and end time of the predicted action instance in the pth video, respectively. represents the predicted action category of the i-th action instance in the p-th video.
[0084] Step 3: Design the loss function of the weakly supervised temporal action localization network for category incremental learning based on feature transfer; the specific process of step 3 is as follows:
[0085] The total loss Los is composed of the classification loss L class And cosine similarity loss L cos and adversarial loss L adv composition:
[0086] Los=L class +L cos +L adv (1)
[0087]
[0088] in, and They represent the true value label and prediction result of the p-th video belonging to the c-th action in task n, respectively. n represents the total number of action categories under task n, mean() represents the averaging function, t is the true value label of the feature task source of the discriminator, and σ is the predicted value of the feature task source of the discriminator.
[0089] Step 4: Set the network model training parameters, as shown in Table 5, and train the sample set T ra S and the corresponding L Tra The networks are sent in batches to the designed feature transfer-based category incremental learning weakly supervised temporal action localization network for training. When the objective function Los converges or reaches the set number of iterations, the network training ends and the network model M is output. The specific process is as follows:
[0090] Step 4.1, set the network model training parameters, that is, set the learning rate variable l, the maximum number of training iterations variable Max_Epoch, the size of each batch of data variable Batch_Size, the minimum change value of the target loss function ΔMini_Loss, the number of training iterations variable step, and initialize step = 1;
[0091] Step 4.2: Input the training video sample set T ra S, true value label training set L Tra ;
[0092] Step 4.3, from the training video sample set T ra S and the true value label training set L Tra Randomly read Batch Size training samples and the true value label The read data is sent to the network constructed in step 2 for network training. In the training process of the discriminator D, the gradient reversal strategy is used, that is, the gradient reversal layer. The gradient reversal layer multiplies the back-propagated gradient by -λ. λ changes from 0 to 1 as the number of training generations increases. p represents the ratio of the current generation setp to the total training generation Max_Epoch;
[0093] Step 4.4, calculate the absolute difference diff of the loss function Los of two iterations during the network training process, and judge whether the training model converges according to diff. If diff < ΔMini_Loss or step > Max_Epoch, the model converges, otherwise the model does not converge;
[0094] Step 4.5: If the model converges, go to step 4.6; if the model does not converge, step = setp + 1, use the Adam optimizer to reversely correct the weight coefficients of each network layer in the network model, continue training, and return to step 4.3;
[0095] Step 4.6, the network training is completed, and the training model M of the end-to-end network is saved;
[0096] Table 5 Network model training parameters
[0097] parameter Parameter Description Value l Learning Rate 0.00001 Max_Epoch Maximum number of training iterations 200 Batch_Size The size of each batch of data 1 Mini_Loss Minimum change value of objective loss function 0.005 step Initial value of training iteration number variable 1
[0098] Step 5: In the test video set T es The model M is tested on S to obtain the action localization result of the video and evaluate the localization result;
[0099] The specific process of step 5 is as follows:
[0100] Step 5.1: Input the test sample set T es S, action truth label test set L Tes And the trained network model M;
[0101] Step 5.2: Define the time sequence action positioning result set variable R e , sequential action positioning result variable R er , test video sample set T es S's total sample number variable R, test sample number counter variable r, R e Initialized to empty, that is, R e = NULL, r is initialized to 1, r = 1;
[0102] Step 5.3: Take the test video sample set T es The rth video in S is sent to M to output the temporal action localization result set R er ;like Figure 4 As shown, the specific process of step 5.3 is:
[0103] (1) Define the task number variable nn and initialize nn = N;
[0104] (2) Take the test video sample set T esThe rth video in S is fed into the Nth feature extractor FE N , get the video features
[0105] (3) Video features Feed into the Nth classifier Cla N , and get the classification results represents the probability score that the action belongs to the i-th category, 1 <i<c N , c N Represents the Nth classifier Cla N The total number of predictable action categories;
[0106] (4) Find the maximum value and the corresponding category number. max() represents the maximum value function;
[0107] (5) If Enter the positioning and post-processing module, select video clips with scores higher than the preset threshold, merge continuous clips into a positioning result and output the positioning result, exit the test, otherwise enter the Nth feature migration module Get the post-migration features thre is the preset threshold, thre = 0.6;
[0108] (6) nn = nn-1, the migrated features As input, go to step (3).
[0109] Step 5.4, R er Append to R e Middle, that is, R e + = R er ;
[0110] Step 5.5, determine whether the number of r in the current test video is greater than or equal to R. If so, the test ends and outputs R e , otherwise r=r+1, return to step 5.3;
[0111] Step 5.6: The positioning results of all test videos are evaluated by calculating the mean average precision mAP to obtain the evaluation result Pa. The calculation formula is shown in (5):
[0112] Pa=mAP(R e ,L Tes ) (5)
[0113] Among them, mAP(*,*) is the mean average precision calculation function, R e and LTes Represent the test video sample set T es S’s temporal action localization result set and the corresponding true value label test set.
[0114] Example 1
[0115] The present invention provides a weakly supervised temporal action localization method based on category incremental learning of feature migration, and uses standard data sets THUMOS14 data set and ActivityNet1.3 data set to test the method of the present invention.
[0116] Dataset description: (1) THUMOS14 dataset: THUMOS14 is a commonly used dataset for temporal action detection published by Idrees et al. in 2014. When training locally, 413 videos containing action instances are selected, and the video lengths range from a few seconds to several minutes. The dataset contains 20 categories, including 200 training videos and 213 test videos with detailed time annotations. The present invention adopts the same training and testing scale as other classical methods, using 200 training videos to train the model of the present invention and using 213 test videos to evaluate it. This dataset has a large number of action instances with shorter durations. In addition, many videos contain multiple action instances for the same action class, making it extremely challenging to accurately locate and identify actions.
[0117] (2) ActivityNet1.3 dataset: ActivityNet dataset is a large dataset for video action recognition and temporal action detection released by Heilbron et al. in 2015. There are two versions of the dataset, ActivityNet1.2 and ActivityNet1.3. The present invention uses ActivityNet1.3 to evaluate the model effect. ActivityNet 1.3 contains 19,994 videos from 200 different activities. The videos are all action videos shot from foreign websites. Its training set has 10,024 videos, the validation set has 4,926 videos, and the test set has 5,044 videos, which contains a total of 200 action categories.
[0118] Algorithm evaluation: Mean Average Precision (mAP): The mean average precision (mAP) under a specific threshold is used as the main evaluation indicator.
[0119] (1) Objective evaluation:
[0120] As shown in Tables 6 and 7 below, the mAP results of the present invention and the previous method RSKP (Weaklysupervised temporal action localization via representative snippet knowledgepropagation proposed by Huang L et al. at CVPR in 2022) under different tIoU on the THUMOS14 and ActivityNet1.3 datasets are respectively shown. The method proposed by the present invention shows excellent results in various performance indicators, which are better than RSKP. The average mAP of the method of the present invention on the THUMOS14 dataset reached 46.90%, and the average mAP of the method of the present invention on ActivityNet reached 25.53%, which can prove the effectiveness of the method of the present invention.
[0121] Table 6 Positioning accuracy under different thresholds on the THUMOS14 dataset
[0122]
[0123] Table 7 Positioning accuracy under different thresholds on the ActivityNet1.3 dataset
[0124]
[0125] Example 2
[0126] Subjective evaluation: The subjective effect diagram of applying the weakly supervised temporal action localization method based on feature migration and category incremental learning in the present invention to the THUMOS14 dataset, as shown in Figure 5-1 to Figure 5-3 As shown, a few frames are randomly sampled from the entire video to represent the entire video clip. Table 8-1, Table 8-2, and Table 8-3 show the real time boundaries and network-localized time boundaries of three action instances on the THUMOS14 dataset. Figure 5-1 The video background is between 0s and 3.1s, and between 23.6s and 32s. Figure 5-1 The corresponding action instance part between 3.1s and 23.6s in Table 8-1 Figure 5-1 The subjective results of the harmonica playing action example are shown. It can be seen that the duration of this example is 20.5 seconds. The action segment and the background segment are relatively obvious. For this relatively simple case, the method of the present invention shows good prediction results. The errors of the predicted action start boundary and end boundary are only 0.94 seconds and 1.39 seconds. Figure 5-2The video background is between 0s and 31.2s, between 39.5s and 40.3s, between 46.4s and 61.9s, and between 69.7s and 86s. Figure 5-2 The corresponding action instance parts are between 31.2s and 39.5s, between 40.3s and 46.4s, and between 61.9s and 69.7s. Table 8-2 corresponds to Figure 5-2 The subjective results of the javelin throwing action example are shown. It can be seen from the figure that the action example is short in duration and is divided into three segments by background segments, which is a relatively complicated situation. The absolute values of the deviations of the start times of the three actions located by the method of the present invention compared to the true values are 1.01s, 2.48s and 2.49s, respectively, and the absolute values of the deviations of the end times of the three actions compared to the true values are 2.24s, 2.29s and 0.4s, respectively. From the positioning results, it can be seen that the category incremental learning module introduced by the method of the present invention eliminates the interference of background segments to a certain extent and enhances the action segments, thereby successfully distinguishing the middle action segments and achieving good positioning results. Figure 5-3 The video background is between 0s and 11.4s, between 12.2s and 18.6s, and between 20.8s and 30s. Figure 5-3 The action instance part between 11.4s and 12.2s, and between 18.6s and 20.8s corresponds to Table 8-3 Figure 5-3 The subjective results of the cricket bowling action example are shown. From the positioning results, it can be seen that the method of the present invention has a good positioning effect. The absolute values of the deviations of the start times of the two actions located by the method of the present invention compared with the true values are 1.07s and 2.02s respectively, and the absolute values of the deviations of the end times of the two actions compared with the true values are 2.01s and 2.15s respectively, which can verify the effectiveness of the method of the present invention.
[0127] Table 8-1 Subjective results of harmonica playing action examples
[0128] Action Category Real time / positioning time of the present invention Action start time Action end time Playing the harmonica Real time / positioning time of the present invention 3.1s / 2.16s 23.6s / 22.21s
[0129] Table 8-2 Subjective results of javelin throwing action examples
[0130] Action Category Real time / positioning time of the present invention Action start time Action end time Javelin Throw1 Real time / positioning time of the present invention 31.2s / 30.19s 39.5s / 37.26s Javelin Throw 2 Real time / positioning time of the present invention 40.3s / 42.78s 46.4s / 44.11s Javelin Throw 3 Real time / positioning time of the present invention 61.9s / 59.41s 69.7s / 70.10s
[0131] Table 8-3 Subjective results of cricket bowling action examples
[0132] Action Category Real time / positioning time of the present invention Action start time Action end time Cricket Bowling 1 Real time / positioning time of the present invention 11.4s / 10.33s 12.2s / 14.21s Cricket Bowling 2 Real time / positioning time of the present invention 18.6s / 16.58s 20.8s / 22.95s
[0133] Example 3
[0134] The class incremental learning weakly supervised temporal action localization method based on feature migration of the present invention is applied to the ActivityNet1.3 dataset. Figure 6-1 to Figure 6-3 As shown, a few frames are randomly sampled from the entire video to represent the entire video clip. Table 9-1, Table 9-2, and Table 9-3 show the subjective results on the ActivityNet1.3 dataset. The duration of the action instances in the ActivityNet1.3 dataset is longer than that of the THUMOS14 dataset, which tests the model's ability to model long distances. Figure 6-1 The video background is between 0s and 14.2s and between 27.7s and 35s. Figure 6-1 The corresponding action instance part between 14.2s and 27.7s in Table 9-1 Figure 6-1 What is demonstrated is a calf lassoing movement in a western cowboy competition. The movement example is relatively complex, but its movement boundary is relatively obvious, so the method of the present invention has a good positioning result. Figure 6-2 The video background is between 0s and 63.4s and between 138.3s and 150s. Figure 6-2 The corresponding action instance part between 63.4s and 138.3s in Table 9-2 Figure 6-2 An action example of installing a carpet is demonstrated. The duration of this action example is 80 seconds, which is a relatively challenging long action situation. From the positioning results, it can be seen that the method of the present invention locates the correct boundary more accurately, proving the effectiveness of the method proposed in the present invention for long time series modeling. Figure 6-3 The video background is between 0s and 4.7s, between 17.6s and 24.4s, and between 35.0s and 40s. Figure 6-3 The corresponding action instance part is between 4.7s and 17.6s, and between 24.4s and 35.0s. Table 9-3 corresponds to Figure 6-3 The beach soccer action example is shown. The example is divided into two segments by the background segment, but the separated action segment and background segment last for a long time and are relatively easy to distinguish. From the positioning results, it can be seen that the method of the present invention correctly locates the two examples.
[0135] Table 9-1 Subjective results of the calf lasso action example
[0136] Action Category Real time / positioning time of the present invention Action start time Action end time Calf Lasso Real time / positioning time of the present invention 14.2s / 12.58s 27.7s / 28.34s
[0137] Table 9-2 Subjective results of carpet installation action example
[0138] Action Category Real time / positioning time of the present invention Action start time Action end time Installing Carpet True value / positioning time of the present invention 63.4s / 64.75s 138.3s / 138.12s
[0139] Table 9-3 Subjective results of beach soccer action examples
[0140] Action Category Real time / positioning time of the present invention Action start time Action end time Beach Soccer 1 True value / positioning time of the present invention 4.7s / 3.41s 17.6s / 18.26s Beach Soccer 2 True value / positioning time of the present invention 24.4s / 22.39s 35.0s / 34.17s
Claims
1. A weakly supervised temporal action localization method based on feature transfer and category incremental learning. Features: The specific steps include: Step 1, obtain a video data set and a corresponding true value label set, and preprocess the video data set, divide the preprocessed video data set into a training video set and a test video set, and divide the corresponding true labels after the preprocessing of the video data set into a training set true value label and a test set true value label; Step 2: Construct a category incremental learning weakly supervised temporal action localization network structure FTIL based on feature transfer; Step 3, design the network loss function Los; Step 4, training the network model M; Step 5: Test model M on the test video set to obtain the action localization results of the video and evaluate the localization results.
2. According to claim 1, the method for weakly supervised temporal action localization based on incremental learning of categories and feature migration, Features: The specific process of step 1 is as follows: Download the action video set VDS from the official website of temporal action detection. 1 , …, V m , …, V M } and the corresponding action truth label set LS = {Y 1 , …, Y m , …, Y M },V m and Y m represents the mth video in VDS and its corresponding action category label, 1≤m≤M, M represents the total number of videos in VDS, W, H, T m Corresponding to V m The width, height and video duration of the image frame. Represents Y m The nth m action instance labels, 1≤n m ≤N m , N m Represents Y m The total number of action instances in ; In the training set, the true value label L Tra middle in, Takes 0 or 1, representing the cth video in the mth video. m Whether the class action exists, C m Indicates the total number of categories; In the test set, the true value label L Tes middle in Respectively The start time, end time and action category number; Divide VDS into training videos T according to the ratio of m:n ra S and test video T es S, Indicates T ra The pth video in S, Indicates T es The qth video in S, P+Q=M, where m>m; Press LS to T ra S and T es S is divided into the training set truth label L Tra And the test set true value label LT es , Indicates L Tra The p-th video truth label in , Indicates L Tes The qth video truth label.
3. According to claim 2, the method for weakly supervised temporal action localization based on incremental learning of categories and feature migration, Features: In step 2, the feature transfer-based category incremental learning weak supervision network structure for temporal action positioning includes a video input module, a task ID generation module, a feature transfer-based adjacent task classifier learning module, and a positioning and post-processing module.
4. According to claim 3, the method for weakly supervised temporal action localization based on incremental learning of categories and feature migration, Features: The specific process of step 2 is: Step 2.1, Video Input Module: From T ra Extract the pth video from S All clips of and its corresponding label As network input, 1 <p<P; Step 2.2, Task ID Generation Module: and the corresponding labels As input to the task ID generation module, we get the training video clips Task ID number n; Step 2.3, adjacent task classifier learning module based on feature transfer: according to the task number n, As input to the nth feature extractor FE n and the n-1th feature extractor FE n-1 In the 13D network, the first to ninth layers are selected as the backbone network for feature extraction, and the output of the ninth layer is used as The deep features F, T and D represent the video length and depth feature dimension respectively. The depth feature F is fed into the nth classifier Cla n , the classifier consists of a series of convolutional layers and a sigmoid layer. n Output classification results represents the probability score that the action belongs to the i-th category, 1 <i<c n , c n Represents the nth classifier Cla n The total number of predictable action categories; Step 2.4, Positioning and post-processing module: The video is sent as input to the positioning and post-processing module, and the video clips with scores higher than the preset threshold are selected, and the continuous clips are merged into a positioning result. and denote the start time and end time of the predicted action instance in the pth video, respectively. represents the predicted action category of the i-th action instance in the p-th video.
5. According to claim 4, the method for weakly supervised temporal action localization based on incremental learning of categories based on feature migration, Features: In step 2.2, The process of generating task ID numbers is as follows: the total number of task ID numbers is set to N, the total number of category tag numbers is set to H, and the corresponding linear table L between category tag numbers and task ID numbers is manually established according to the tag information corresponding to the category tag number. The action category tag number h is input, and the linear table query function LocatedElem (Lh) is called. The corresponding linear table L can be queried in table L. The task ID number n.
6. According to claim 5, the method for weakly supervised temporal action localization based on incremental learning of categories and feature migration, Features: In step 2.3, the adjacent task classifier learning module based on feature migration includes feature extractors, feature migration modules and classifiers under different tasks, and the specific process is as follows: (1) Get the current fragment Task ID number n; (2) Determine whether n is equal to 1. If so, put the fragment Then, they are fed into the first feature extractor FE. 1 , the first classifier Cla 1 , and get the output Indicates the probability score that the video action belongs to the xth category, 1 <x<c 1 , c 1 Represents the first classifier Cla 1 The total number of predictable action categories, proceed to step 2.4; (3) Otherwise, put the fragment Feed into the nth and n-1th feature extractors FE n and FE n-1 , and get the corresponding features and The superscript n indicates the corresponding feature extractor number, and the subscript n corresponds to the task number. Feed into the nth classifier Cla n , output represents the probability score that the action belongs to the i-th category, 1 <i<c n , c n Represents the nth classifier Cla n The total number of predictable action categories, go to step 2.4, and then Send to the nth feature migration module Output post-migration features Bundle and are fed into the generator G output and Then and They are sent to the discriminator D respectively, and the output task source prediction result σ, σ = (ρ n , ρ n-1 ), ρ n and ρ n-1 They represent the probability values of the feature source belonging to task n and task n-1 respectively.
7. The method for weakly supervised temporal action localization based on incremental learning of categories and feature migration according to claim 6, Features: In step 3, the total loss Los includes the classification loss L class , cosine similarity loss L cos and adversarial loss L adv , as follows: The=L class +L cos +L adv (1) in, and They represent the true value label and prediction result of the p-th video belonging to the c-th action in task n, respectively. n represents the total number of action categories under task n, mean() represents the averaging function, t is the true value label of the feature task source of the discriminator, and σ is the predicted value of the feature task source of the discriminator.
8. The method for weakly supervised temporal action localization based on incremental learning of categories and feature migration according to claim 7, Features: The specific process of training the network model M in step 4 is as follows: Step 4.1, set the network model training parameters, that is, set the learning rate variable l, the maximum number of training iterations variable Max_Epoch, the size of each batch of data variable Batch_Size, the minimum change value of the target loss function ΔMini_Loss, the number of training iterations variable step, and initialize step = 1; Step 4.2: Input the training video sample set T ra S, true value label training set L Tra ; Step 4.3, from the training video sample set T ra S and the true value label training set L Tra Randomly read Batch Size training samples and the true value label The read data is sent to the network constructed in step 2 for network training. In the training process of the discriminator D, the gradient reversal strategy is used, that is, the gradient reversal layer. The gradient reversal layer multiplies the back-propagated gradient by -λ. λ changes from 0 to 1 as the number of training generations increases. p represents the ratio of the current generation step to the total training generation Max_Epoch; Step 4.4, calculate the absolute difference diff of the loss function Los of two iterations during the network training process, and judge whether the training model converges according to diff. If diff < ΔMini_Loss or step > Max_Epoch, the model converges, otherwise the model does not converge; Step 4.5: If the model converges, go to step 4.6; if the model does not converge, step = step + 1, use the Adam optimizer to reversely correct the weight coefficients of each network layer in the network model, continue training, and return to step 4.3; Step 4.6, network training is completed and the training model M of the end-to-end network is saved.
9. The method for weakly supervised temporal action localization based on incremental learning of categories and feature migration according to claim 8, Features: The specific process of step 5 is as follows: Step 5.1: Input the test video sample set T es S, true value label test set L Tes And the trained network model M; Step 5.2: Define the time sequence action positioning result set variable R e , sequential action positioning result variable R er , test video sample set T es S's total sample number variable R, test sample number counter variable r, R e Initialized to empty, that is, R e = NULL, r is initialized to 1, r = 1; Step 5.3: Take the test video sample set T es The rth video in S is sent to M to output the temporal action localization result set R er ; Step 5.4, R er Append to R e Middle, that is, R e + = R er ; Step 5.5, determine whether the number of r in the current test video is greater than or equal to R. If so, the test ends and outputs R e , otherwise r=r+1, return to step 5.3; Step 5.6: The positioning results of all test videos are evaluated by calculating the mean average precision mAP to obtain the evaluation result Pa. The calculation formula is shown in (6): Pa=mAP(R e ,L Tes ) (6) Among them, mAP(*), *) is the mean average precision calculation function, R e and L Tes Represent the test video sample set T es S’s temporal action localization result set and the corresponding true value label test set.
10. The method for weakly supervised temporal action localization based on incremental learning of categories and feature migration according to claim 9, Features: The specific process of step 5.3 is as follows: (1) Define the task number variable nn and initialize nn = N; (2) Take the test video sample set T es The rth video in S is fed into the Nth feature extractor FE N , get the video features (3) Video features Feed into the Nth classifier Cla N , and get the classification results represents the probability score that the action belongs to the i-th category, 1 <i<c N , c N Represents the Nth classifier Cla N The total number of predictable action categories; (4) Find the maximum value and the corresponding category number. max() represents the maximum value function; (5) If Enter the positioning and post-processing module, select video clips with scores higher than the preset threshold, merge continuous clips into a positioning result and output the positioning result, exit the test, otherwise enter the Nth feature migration module Get the post-migration features thre is the preset threshold; (6) nn = nn-1, the migrated features As input, go to step (3).
