A single-frame supervised temporal action detection method based on high-quality pseudo-labels
By building a seed nomination generation model, a nomination package generation module and a refined network, high-quality pseudo-labels are obtained and fully supervised timing action detectors are trained, and the problems of insufficient quality and low time efficiency in the existing technology are solved, and the performance of single-frame supervised timing action detection is significantly improved.
Patent Information
- Application Number
- CN202411014227.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2044-07-26
AI Technical Summary
The existing single-frame supervision timing action detection method has shortcomings in improving pseudo-label quality and time efficiency, which makes it difficult to balance detection performance and labeling costs.
By building a seed nomination generation model, a nomination package generation module and a refined network, high-quality pseudo-labels are obtained, and these pseudo-labels are used to train a fully supervised timing action detector, which significantly improves detection performance.
It significantly improves the performance of single-frame supervision timing action detection, reduces labeling costs, simplifies the detection process, and avoids the complexity of multi-stage prediction.
Smart Images
Figure CN118823648B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision, and in particular to a single-frame supervised temporal action detection method based on high-quality pseudo-labels. Background Art
[0002] With the development of informatization, the number of videos on the Internet has increased significantly. How to fully analyze existing videos is crucial for video websites. By detecting the temporal motion of videos, we can effectively reduce labor costs and complete tasks such as sports video analysis, TV series intelligent editing, and video secondary creation. At present, the huge amount of video data requires a lot of labor costs to complete the annotation of temporal motion detection tasks. There is an urgent need for effective weakly supervised temporal motion detection methods to reduce the annotation cost while ensuring detection performance.
[0003] Single-frame supervision is a temporal action detection annotation method that can balance annotation cost and detection performance. Existing single-frame supervised temporal action detection mainly adopts two methods: "classification first, then detection" and "pseudo-labeling". The "pseudo-labeling" based method has attracted widespread attention in recent years due to its better detection performance. However, the existing "pseudo-labeling" method usually directly generates pseudo-labels with the help of the existing "classification first, then detection" method, while ignoring how to further improve the quality of pseudo-labels. At the same time, the existing technology usually requires multi-stage prediction to achieve temporal action detection, which affects the time efficiency of its application. Summary of the invention
[0004] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a single-frame supervised temporal action detection method based on high-quality pseudo-labels, in order to obtain high-quality pseudo-labels, thereby significantly improving the single-frame supervised temporal action detection performance.
[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:
[0006] The single-frame supervised temporal action detection method based on high-quality pseudo-labels of the present invention is characterized in that it comprises the following steps:
[0007] Step 1: Obtain video datasets and single-frame supervision, and extract spatiotemporal features of the video;
[0008] Step 1.1: Get a video in the video dataset ,in, Indicates t Video clips, T Indicates the total number of video clips;
[0009] Through the pre-trained feature extractor Extract and obtain the t-th spatiotemporal feature, thereby obtaining the spatiotemporal feature vector of V ; D represents the dimension of spatiotemporal features;
[0010] Step 1.2: Label the action instances in the video V and obtain single-frame supervision ,in, represents the single-frame supervision of the i-th action instance in video V, represents the timestamp position of the i-th action instance in the video V, represents the action category of the i-th action instance, and C represents the total number of action categories; is the number of action instances in video V;
[0011] When the action category of the i-th action instance is the c-th, let , otherwise, let ;
[0012] Will After the summation, the action categories corresponding to the positions greater than 1 in the summation result are set to 1, and the action categories corresponding to the remaining positions are set to 0, thereby obtaining the action category labeling of the video V ;
[0013] Step 2: Build a seed nomination generation model and perform Processing is performed to obtain a list of nominations for all action categories. and seed nomination list , thereby training the seed nomination generation model and outputting the optimal seed nomination list ;
[0014] Step 3: Build a nomination package generation module and use it to Processing is performed to obtain the nominated package list B;
[0015] Step 4: Build a refined network and use it to process B to obtain high-quality pseudo labels , thereby training the refinement network and obtaining high-quality pseudo labels output by the trained refinement network ;
[0016] Step 5: Use high-quality pseudo labels as supervision information to train a fully supervised temporal action detector to obtain a trained fully supervised temporal action detector, which is used to predict any video to obtain a temporal action detection result.
[0017] The single-frame supervised temporal action detection method based on high-quality pseudo-labels described in the present invention is also characterized in that the seed nomination generation model in step 2 includes: a one-dimensional convolution layer, a ReLU activation function, a random inactivation layer, a one-dimensional convolution layer, and a sigmoid() function;
[0018] Step 2.1: Input the spatiotemporal feature vector F into the seed nomination box generation model for processing to obtain the action category prediction value on all video clips. ;
[0019] Apply the threshold method to the prediction value of the action category c in the action category prediction value O Processing is performed to filter out all video clips above the threshold, and continuous video clips are combined to obtain all nominations for the cth action category, and then a nomination list of all action categories is obtained. ;in, represents all predicted values of column c in O; represents the u-th nomination in the nomination list, They represent the starting position, ending position, action category and confidence score of the u-th nomination respectively. represents the total number of nominations generated; the confidence score is the average of the predicted probabilities of the action categories of all video clips in the corresponding nomination;
[0020] Step 2.2, generating a seed nomination frame;
[0021] For single frame supervision The i-th single frame supervision in ,like The action category of is the cth, then from Select the action category c, and the start position and end position contain All nominations, and the nomination with the highest confidence score is used as the seed nomination for the i-th single frame supervision; thus obtaining the seed nomination list ,in, represents the i-th seed nomination, Respectively represent the starting position and ending position of the i-th seed nomination;
[0022] Step 2.3: Construct the cross entropy loss function through formula (1) , and is used to perform back propagation training on the seed nomination generation model to obtain the trained seed nomination generation model and the optimal seed nomination list output by it ,in, represents the i-th best seed nomination, Respectively represent the starting position and ending position of the i-th best seed nomination;
[0023] (1)
[0024] In formula (1), is the vector obtained by averaging O along the first dimension, and express and The value of the cth dimension in .
[0025] The nomination package generation module in step 3 is composed of a scale propagation unit and a center position propagation unit;
[0026] Step 3.1: The scale propagation unit uses a scale factor array List of nominations for best seeds Each optimal seed nomination in is scaled and propagated, so that the starting position of the i-th optimal seed nomination after the j-th scale factor propagation is obtained using formula (2): , End position ,in, represents the number of scale propagation factors, represents the jth scale factor:
[0027] (2)
[0028] In formula (2), represents the center position of the i-th best seed nomination, represents the length of the i-th best seed nomination, and has:
[0029] (3)
[0030] Step 3.2: The center position propagation unit uses the center position factor array For each optimal seed nomination after scale propagation, center position propagation is performed, so that formula (4) obtains the starting position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor propagation: , End position , represents the number of central location factors, Represents the kth center position factor:
[0031] (4)
[0032] In formula (4), and are the center position and length of the i-th optimal seed nomination after the j-th scale factor propagation, respectively;
[0033] Step 3.3: Nominate the best seed list After applying all scale factors and center position factors in turn, we get the list of nominated packages. ,in, The nomination package obtained after the i-th seed nomination is first propagated through the scale and then propagated through the center position; and They respectively represent the starting position and ending position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor.
[0034] The refinement network in step 4 is composed of a selection module and a sorting module, and is used to extract high-quality pseudo labels from the nomination package;
[0035] Step 4.1: The selection module uniformly samples the nominated package list B based on F to obtain a time series length of , and the RoI feature with dimension D ,in, represents the number of nominations in each nomination package, and ;
[0036] Will The input is fed into two shared fully connected layers and then fed into two parallel fully connected layers for calculation, and two prediction results are obtained accordingly. The two prediction results are processed using the softmax() function in the second and third dimensions respectively, and the corresponding dimensions are Two contribution matrices and , so that the selection module finally outputs the classification confidence of the nominated package list B ; represents the Hadamard product;
[0037] Step 4.2: The sorting module uses two fully connected layers to sort the RoI features. Processing to obtain embedded features ;Will After averaging along the second dimension, the average result is the same as Add together to obtain enhanced features; after the enhanced features are predicted by a fully connected layer, the temporal confidence of the nominated package list B is finally output ;
[0038] Step 4.3: Calculate the nomination confidence , where bdc means The last dimension of is broadcast to the C dimension;
[0039] based on Select the first k nominations with the highest nomination confidence from each nomination package in the nomination package list B, and perform weighted average on the start and end positions of the k nominations, so as to obtain a high-quality pseudo label denoted as ,in, represents the pseudo label extracted from the i-th nominated bag, are the starting and ending positions of the pseudo-labels extracted from the i-th nomination bag;
[0040] Step 4.4: Use formula (5) to construct the cross entropy loss function of the selection module ;
[0041] (5)
[0042] In formula (5), for The result of summing along the second dimension, express The prediction result of the i-th nominated bag in the c-th action category;
[0043] The intersection-over-union ratio between all nominations in each nomination package and its corresponding seed nomination is used as the true label; thereby constructing the regression loss function of the ranking module based on the square of the difference between the temporal confidence and the true label;
[0044] Based on the cross entropy loss function The refined network is trained by back propagation using a regression loss function, thereby obtaining a trained refined network and its output high-quality pseudo labels. ,in, represents the high-quality pseudo-label extracted from the i-th nominated bag, are the starting and ending positions of the high-quality pseudo labels extracted from the i-th nominated bag, respectively.
[0045] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the single-frame supervisory timing motion detection method, and the processor is configured to execute the program stored in the memory.
[0046] The present invention provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program, which is characterized in that when the computer program is run by a processor, the steps of the single-frame supervisory sequential action detection method are executed.
[0047] Compared with the prior art, the present invention has the following beneficial effects:
[0048] 1. The present invention constructs a nomination package for each single-frame supervision efficiently by constructing a seed nomination generation model and a nomination package generation module. The optimal seed nomination output by the seed nomination generation model can provide the approximate location information of the real action instance for the single-frame supervision. Furthermore, the nomination package generation module obtains a nomination package containing multiple nominations through scale propagation and center position propagation based on the optimal seed nomination. In the nomination package, there are nominations that are closer to the real action instance in terms of temporal position, thereby providing a guarantee for the subsequent generation of high-quality pseudo-labels.
[0049] 2. The present invention ensures the extraction of high-quality pseudo labels by constructing a refined network. Furthermore, the performance of single-frame supervised sequential action detection is significantly improved by training the sequential action detector based on the high-quality pseudo labels.
[0050] 3. The application of the present invention is more convenient and concise. When detecting action instances in new videos, only the trained temporal action detector is required, and there is no need to perform a series of steps for generating pseudo labels again, thus solving the problem that the prior art requires multi-stage prediction.
[0051] 4. The method of the present invention has higher flexibility, and the high-quality pseudo labels generated by it can be used to train all temporal action detectors. Therefore, more powerful temporal action detectors can be fully utilized in the future, further improving the performance of single-frame supervised temporal action detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a flow chart of the single-frame supervised temporal action detection method based on high-quality pseudo-labels of the present invention;
[0053] Figure 2 This is a structural diagram of the network detailed in the present invention. DETAILED DESCRIPTION
[0054] In this embodiment, a single-frame supervised temporal action detection method based on high-quality pseudo labels is provided, which improves the performance of single-frame supervised temporal action detection by extracting high-quality pseudo labels for each single-frame supervision. The method includes: obtaining a video dataset and single-frame supervision, extracting spatiotemporal features of the video, constructing a seed nomination generation model, constructing a nomination package generation module, constructing a refinement network, training a fully supervised temporal action detector, and predicting any video; specifically, Figure 1 As shown, the method includes:
[0055] Step 1: Obtain video datasets and single-frame supervision, and extract spatiotemporal features of the video;
[0056] Step 1.1: Get a video in the video dataset ,in, represents the tth video clip, T represents the total number of video clips;
[0057] Through the pre-trained feature extractor Extract and obtain the t-th spatiotemporal feature, thereby obtaining the spatiotemporal feature vector of V ; D represents the dimension of spatiotemporal features; Generally speaking, I3D pre-trained on the Kinetics-400 dataset is used as the feature extractor, in which case the extracted feature dimension is 2048.
[0058] Step 1.2: Label the action instances in the video V and obtain single-frame supervision ,in, represents the single-frame supervision of the i-th action instance in video V, represents the timestamp position of the i-th action instance in the video V, represents the action category of the i-th action instance, and C represents the total number of action categories; is the number of action instances in video V;
[0059] When the action category of the i-th action instance is the c-th, let , otherwise, let ;
[0060] Will After the summation, the action categories corresponding to the positions greater than 1 in the summation result are set to 1, and the action categories corresponding to the remaining positions are set to 0, thereby obtaining the action category labeling of the video V .
[0061] Step 2: Construct a seed nomination generation model, including: one-dimensional convolution layer, ReLU activation function, random dropout layer, one-dimensional convolution layer, sigmoid() function; in the specific setting, the output dimension of the first one-dimensional convolution layer is set to 2048, the convolution kernel size and step size are set to 3 and 1 respectively; the output dimension of the second one-dimensional convolution layer is set to C , the convolution kernel size and stride are both set to 1; the dropout rate of the random dropout layer is set to 0.5.
[0062] Step 2.1: Transform the spatiotemporal feature vector F Input the seed nomination box generation model for processing to obtain the action category prediction value on all video clips ;
[0063] Apply threshold method to predict action category O The action category is c The predicted value of The threshold method uses multiple thresholds, that is, each value in the array [0.025, 0.05, 0.075, 0.1, 0.125, 0.15, 0.175, 0.2, 0.225, 0.25] is used as the threshold in turn. All video clips above the threshold are screened, and continuous video clips are combined to obtain the action category c All nominations of , and then get the nomination list of all action categories ;in, represents all predicted values of column c in O; represents the u-th nomination in the nomination list, They represent the starting position, ending position, action category and confidence score of the u-th nomination respectively. represents the total number of nominations generated; the confidence score is the average of the predicted probabilities of the action categories of all video clips in the corresponding nomination.
[0064] Step 2.2, generating a seed nomination frame;
[0065] For single frame supervision The i-th single frame supervision in ,like The action category of is the cth, then from Select the action category c, and the start position and end position contain All nominations, and the nomination with the highest confidence score is used as the seed nomination for the i-th single frame supervision; thus obtaining the seed nomination list ,in, represents the i-th seed nomination, They respectively represent the starting position and ending position of the i-th seed nomination.
[0066] Step 2.3: Construct the cross entropy loss function through formula (1) , and is used to perform back propagation training on the seed nomination generation model to obtain the trained seed nomination generation model and the optimal seed nomination list output by it ,in, represents the i-th best seed nomination, Respectively represent the starting position and ending position of the i-th best seed nomination;
[0067] (1)
[0068] In formula (1), is the vector obtained by averaging O along the first dimension, and express and The value of the cth dimension in .
[0069] The nomination package generation module is composed of a scale propagation unit and a center position propagation unit, and is used to Processing is performed to obtain the nominated package list B;
[0070] Step 3.1: The scale propagation unit uses a scale factor array List of nominations for best seeds Each optimal seed nomination in is scaled and propagated, so that the starting position of the i-th optimal seed nomination after the j-th scale factor propagation is obtained using formula (2): , End position ,in, represents the number of scale propagation factors, represents the jth scale factor, where the scale factor array can be set to [0.5, 0.7, 0.9, 1.0, 1.1,1.3, 1.5]:
[0071] (2)
[0072] In formula (2), represents the center position of the i-th best seed nomination, represents the length of the i-th best seed nomination, and has:
[0073] (3)
[0074] Step 3.2: The center position propagation unit uses the center position factor array For each optimal seed nomination after scale propagation, center position propagation is performed, so that formula (4) obtains the starting position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor propagation: , End position , represents the number of central location factors, Represents the kth center position factor, where the center position factor array can be set to [0.1]:
[0075] (4)
[0076] In formula (4), and are respectively the center position and length of the i-th optimal seed nomination after the j-th scale factor is propagated.
[0077] Step 3.3: Nominate the best seed list After applying all scale factors and center position factors in turn, we get the list of nominated packages. ,in, The nomination package obtained after the i-th seed nomination is first propagated through the scale and then propagated through the center position; and They respectively represent the starting position and ending position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor.
[0078] Step 4: Figure 2 As shown, a refinement network is constructed, which is composed of a selection module and a ranking module and is used to extract high-quality pseudo labels from the nomination bag;
[0079] Step 4.1, the selection module uniformly samples the nominated package list B based on F, and obtains a time series length of , and the RoI feature with dimension D ,in, represents the number of nominations in each nomination package, and ; Can be set to 7.
[0080] Will The input is fed into two shared fully connected layers and then fed into two parallel fully connected layers for calculation, and two prediction results are obtained accordingly. The two prediction results are processed using the softmax() function in the second and third dimensions respectively. The softmax() function is used in the second dimension to calculate the contribution of each nomination in the nomination package to the classification of the nomination package as the corresponding action category; the softmax() function is used in the third dimension to calculate the action category probability value of each nomination, and the corresponding dimensions are Two contribution matrices and , so that the selection module finally outputs the classification confidence of the nominated package list B ; Represents the Hadamard product.
[0081] Step 4.2: The sorting module uses two fully connected layers to perform RoI features Processing to obtain embedded features ;Will After averaging along the second dimension, the average result is the same as Add together to obtain enhanced features; after the enhanced features are predicted by a fully connected layer, the temporal confidence of the nominated package list B is finally output ; By By averaging the second dimension, we can get the feature average of all nominations in each nomination. This average is used to avoid the problem of inconsistent labels when different nominations contain nominations at the same temporal position. At the same time, this average makes each nomination pay more attention to the features in the nomination package, which is beneficial to the training of the sorting module.
[0082] Step 4.3: Calculate the nomination confidence , where bdc means The last dimension of is broadcast to the C dimension;
[0083] based on Select the first k nominations with the highest nomination confidence from each nomination package in the nomination package list B, and perform weighted average on the start and end positions of the k nominations. When performing weighted average, the weight is set to the result of applying the softmax() function to the nomination confidence of the k nominations, so as to obtain a high-quality pseudo label denoted as ,in, represents the pseudo label extracted from the i-th nominated bag, are the starting and ending positions of the pseudo-labels extracted from the i-th nominated bag, respectively.
[0084] Step 4.4: Use formula (5) to construct the cross entropy loss function of the selection module ;
[0085] (5)
[0086] In formula (5), for The result of summing along the second dimension, express The prediction result of the i-th nominated bag in the c-th action category.
[0087] Since the real temporal position of each action instance is not fully labeled under single-frame supervision, the intersection-over-union ratio between all nominations in each nomination package and its corresponding seed nomination is used as the real label. Based on the square of the difference between the temporal confidence and the real label, the regression loss function of the ranking module is constructed.
[0088] Based on the cross entropy loss function The refined network is trained by back propagation using a regression loss function, thereby obtaining a trained refined network and its output high-quality pseudo labels. ,in, represents the high-quality pseudo-label extracted from the i-th nominated bag, are the starting and ending positions of the high-quality pseudo labels extracted from the i-th nominated bag, respectively.
[0089] Step 5: Use high-quality pseudo labels as supervision information to train a fully supervised temporal action detector to obtain a trained fully supervised temporal action detector, which is used to predict any video to obtain a temporal action detection result.
[0090] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0091] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.
[0092] In summary, the present invention obtains high-quality pseudo labels for training a fully supervised temporal action detector through a seed nomination generation model, a nomination package generation module, and a refinement network. This innovative method alleviates the problem that existing methods pay little attention to the quality of pseudo labels. At the same time, based on the extracted high-quality pseudo labels, the performance of single-frame supervised temporal action detection is greatly improved. The detection performance comparison results with the existing methods are shown in Table 1:
[0093] Table 1
[0094]
[0095] The mAP in Table 1 is mean Average Precision, that is, the average precision, and the higher the value, the better. In addition, after the training is completed, the present invention only needs to use the trained fully supervised temporal action detector to realize the temporal action detection of any video, which alleviates the problem of complex prediction steps caused by the need for multi-stage prediction in existing methods. For example, the latest method HR-Pro must be based on two-stage prediction to achieve better detection performance. Finally, since the present invention focuses on improving the quality of pseudo-labels, the high-quality pseudo-labels it generates can be applied to any fully supervised temporal action detector. This makes it possible for the performance of single-frame supervised temporal action detection to be improved as the performance of the fully supervised temporal action detector is improved. Compared with existing methods, the present invention is more flexible and convenient.
Claims
1. A single-frame supervised temporal action detection method based on high-quality pseudo-labels, characterized in that: The following steps are involved: Step 1: Obtain video datasets and single-frame supervision, and extract spatiotemporal features of the video; Step 1.1: Get a video in the video dataset ,in, represents the tth video clip, T represents the total number of video clips; Through the pre-trained feature extractor Extract and obtain the t-th spatiotemporal feature, thereby obtaining the spatiotemporal feature vector of V ; D represents the dimension of spatiotemporal features; Step 1.2: Label the action instances in the video V and obtain single-frame supervision ,in, represents the single-frame supervision of the i-th action instance in video V, represents the timestamp position of the i-th action instance in the video V, represents the action category of the i-th action instance, and C represents the total number of action categories; is the number of action instances in video V; When the action category of the i-th action instance is the c-th, let , otherwise, let ; Will After the summation, the action categories corresponding to the positions greater than 1 in the summation result are set to 1, and the action categories corresponding to the remaining positions are set to 0, thereby obtaining the action category labeling of the video V ; Step 2: Build a seed nomination generation model and perform Processing is performed to obtain a list of nominations for all action categories. and seed nomination list , thereby training the seed nomination generation model and outputting the optimal seed nomination list ; Step 3: Build a nomination package generation module and use it to Processing is performed to obtain the nominated package list B; Step 4: Construct a refinement network consisting of a selection module and a sorting module, and use it to process B to obtain high-quality pseudo labels. , thereby training the refinement network and obtaining high-quality pseudo labels output by the trained refinement network ; Step 4.1: The selection module uniformly samples the nominated package list B based on F to obtain a time series length of , and the RoI feature with dimension D ,in, Indicates the number of nominations in each nomination package; Will The input is fed into two shared fully connected layers and then fed into two parallel fully connected layers for calculation, and two prediction results are obtained accordingly. The two prediction results are processed using the softmax() function in the second and third dimensions respectively, and the corresponding dimensions are Two contribution matrices and , so that the selection module finally outputs the classification confidence of the nominated package list B ; represents the Hadamard product; Step 4.2: The sorting module uses two fully connected layers to sort the RoI features. Processing to obtain embedded features ;Will After averaging along the second dimension, the average result is the same as Add together to obtain enhanced features; after the enhanced features are predicted by a fully connected layer, the temporal confidence of the nominated package list B is finally output ; Step 4.3: Calculate the nomination confidence , where bdc means The last dimension of is broadcast to the C dimension; based on Select the first k nominations with the highest nomination confidence from each nomination package in the nomination package list B, and perform weighted average on the start and end positions of the k nominations, so as to obtain a high-quality pseudo label denoted as ,in, represents the pseudo label extracted from the i-th nominated bag, are the starting and ending positions of the pseudo-labels extracted from the i-th nomination bag; Step 4.4: Use formula (5) to construct the cross entropy loss function of the selection module ; (5) In formula (5), for The result of summing along the second dimension, express The prediction result of the i-th nominated bag in the c-th action category; The intersection-over-union ratio between all nominations in each nomination package and its corresponding seed nomination is used as the true label; thereby constructing the regression loss function of the ranking module based on the square of the difference between the temporal confidence and the true label; Based on the cross entropy loss function The refined network is trained by back propagation using a regression loss function, thereby obtaining a trained refined network and its output high-quality pseudo labels. ,in, represents the high-quality pseudo-label extracted from the i-th nominated bag, are the starting and ending positions of the high-quality pseudo labels extracted from the i-th nominated bag, respectively; Step 5: Utilize high-quality pseudo-labels As supervision information, a fully supervised temporal action detector is trained to obtain a trained fully supervised temporal action detector, which is used to predict any video to obtain a temporal action detection result.
2. According to the single-frame supervised temporal action detection method based on high-quality pseudo-labels according to claim 1, it is characterized in that: The seed nomination generation model in step 2 includes: a one-dimensional convolution layer, a ReLU activation function, a random inactivation layer, a one-dimensional convolution layer, and a sigmoid() function; Step 2.1: Input the spatiotemporal feature vector F into the seed nomination generation model for processing to obtain the action category prediction value on all video clips. ; Apply the threshold method to the prediction value of the action category c in the action category prediction value O Processing is performed to filter out all video clips above the threshold, and continuous video clips are combined to obtain all nominations for the cth action category, and then a nomination list of all action categories is obtained. ;in, represents all predicted values of column c in O; represents the u-th nomination in the nomination list, They represent the starting position, ending position, action category and confidence score of the u-th nomination respectively. represents the total number of nominations generated; the confidence score is the average of the predicted probabilities of the action categories of all video clips in the corresponding nomination; Step 2.2, generating a seed nomination frame; For single frame supervision The i-th single frame supervision in ,like The action category of is the cth, then from Select the action category c, and the start position and end position contain All nominations, and the nomination with the highest confidence score is used as the seed nomination for the i-th single frame supervision; thus obtaining the seed nomination list ,in, represents the i-th seed nomination, Respectively represent the starting position and ending position of the i-th seed nomination; Step 2.3: Construct the cross entropy loss function through formula (1) , and used to perform back propagation training on the seed nomination generation model to obtain the trained seed nomination generation model and the optimal seed nomination list output by it ,in, represents the i-th best seed nomination, Respectively represent the starting position and ending position of the i-th best seed nomination; (1) In formula (1), is the vector obtained by averaging O along the first dimension, and express and The value of the cth dimension in .
3. According to the single-frame supervised temporal action detection method based on high-quality pseudo-labels according to claim 2, it is characterized in that: The nomination package generation module in step 3 is composed of a scale propagation unit and a center position propagation unit; Step 3.1: The scale propagation unit uses a scale factor array List of nominations for best seeds Each optimal seed nomination in is scaled and propagated, so that the starting position of the i-th optimal seed nomination after the j-th scale factor propagation is obtained using formula (2): , End position ,in, represents the number of scale propagation factors, represents the jth scale factor: (2) In formula (2), represents the center position of the i-th best seed nomination, represents the length of the i-th best seed nomination, and has: (3) Step 3.2: The center position propagation unit uses the center position factor array For each optimal seed nomination after scale propagation, center position propagation is performed, so that formula (4) obtains the starting position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor propagation: , End position , represents the number of central location factors, Represents the kth central position factor: (4) In formula (4), and are the center position and length of the i-th optimal seed nomination after the j-th scale factor propagation, respectively; Step 3.3: Nominate the best seed list After applying all scale factors and center position factors in turn, we get the list of nominated packages. ,in, The nomination package obtained after the i-th seed nomination is first propagated through the scale and then propagated through the center position; and They respectively represent the starting position and ending position of the i-th optimal seed nomination after the j-th scale factor propagation of the k-th center position factor.
4. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the single-frame supervision temporal motion detection method described in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the single-frame supervised temporal action detection method described in any one of claims 1-3 are executed.