Low-pass filtering method for temporal positioning of actions

Abstract

A low-pass filtering method for temporal axis positioning of an action, step 1: using a spatiotemporal feature extractor to extract features of the entire input video; step 2: using a related action proposal algorithm to calculate an action proposal based on the extracted features of the entire input video; step 3: using a low-pass filter to filter out some high-frequency information for the features corresponding to each action proposal; step 4: using temporal downsampling to calculate features corresponding to the action proposal; and step 5: using the features corresponding to the action proposal to calculate the final position and category corresponding to the action. The method of the present invention uses low-pass filtering to remove the aliasing problem existing in the temporal axis positioning model of the action, and by dynamically adjusting the cutoff frequency, it also preserves some important high-frequency information, thereby effectively improving the detection performance of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of video analysis, in particular to a low-pass filtering method for temporal axis positioning of an action. Background Art

[0002] The temporal location of actions is an important task in the field of computer vision. Its main focus is on detecting the actions in the video. Currently, this technology is mainly implemented with the help of deep neural networks. It takes the video as input and the start position, end position and category of the action in the video as output. The general steps include:

[0003] Step 1. Extract video features;

[0004] Step 2. Predict action proposals based on video features. Here, action proposals refer to preliminary action predictions.

[0005] Step 3. Calculate the features corresponding to the action proposal based on the position of the action proposal and the video features, and further predict the final position and category of the action instance.

[0006] In most action timeline localization models, step 1 is generally completed by a spatiotemporal feature extractor (such as TSN); step 2 can be implemented in two ways: single-stage and double-stage. The single-stage method generally uses pre-set anchor boxes with different positions and shapes as action proposals; the double-stage method uses some classic algorithms to calculate the action proposals. Step 3 inputs the features of the action proposals into subsequent classifiers and regressors to predict the position and category of the action.

[0007] In the traditional temporal location model of actions, temporal downsampling techniques (such as average pooling, maximum pooling, and average sampling) are widely used in the process of extracting action proposal features in step 2. However, these temporal downsampling techniques are often simple in design and lack consideration of the relationship between the downsampling coefficient and the original signal frequency, which often causes aliasing problems in the frequency domain. Summary of the invention

[0008] The low-pass filtering method for time axis positioning of actions of the present invention uses low-pass filtering to remove the aliasing problem existing in the time axis positioning model of actions, and by dynamically adjusting the cutoff frequency, also preserves some important high-frequency information, thereby effectively improving the detection performance of the model.

[0009] The technical solution of the present invention is as follows:

[0010] The low-pass filtering method for temporal axis positioning of an action of the present invention comprises the following steps: step 1: using a spatiotemporal feature extractor to extract features of the entire input video; step 2: using a related action proposal algorithm to calculate an action proposal based on the extracted features of the entire input video; step 3: using a low-pass filter to filter out part of the high-frequency information of the features corresponding to each action proposal; step 4: using temporal downsampling to calculate features corresponding to the action proposal; and step 5: using the features corresponding to the action proposal to calculate the final corresponding position and category of the action.

[0011] Preferably, in the above-mentioned low-pass filtering method for temporal positioning of actions, in step 3, part of the high-frequency information is a high-frequency passband of the regional features corresponding to the action proposal.

[0012] Preferably, in the above-mentioned low-pass filtering method for temporal positioning of actions, in step 5, the features corresponding to the action proposal in step 4 are input into the classifier and regressor, the final position and category of the action are calculated, and the final predicted action result is output.

[0013] According to the technical solution of the present invention, the beneficial effects produced are:

[0014] The present invention inserts a low-pass filter before the time series downsampling operation (such as maximum pooling, mean sampling, etc.) to filter out part of the high-frequency information, thereby removing the aliasing problem existing in the time axis positioning model of the action; at the same time, the present invention is designed to learn different truncation frequencies for different action proposals to adjust the relationship between de-aliasing and preserving high-frequency information, so that the low-pass filter can preserve as much high-frequency information as possible while removing the aliasing, thereby avoiding the degradation of model detection performance due to the removal of too much high-frequency information.

[0015] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0017] Figure 1 A flow chart of a low-pass filtering method for temporal positioning of an action according to the present invention;

[0018] Figure 2a and Figure 2b is the spectrum difference between the action proposal feature before sampling and the action proposal feature after sampling, where Figure 2a is the aliasing error heat map without using low-pass filter; Figure 2bis a heat map of aliasing errors after applying a low-pass filter. DETAILED DESCRIPTION

[0019] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0020] The present invention alleviates the aliasing problem by inserting a low-pass filter before the time-series downsampling operation. However, the low-pass filter will filter out the high-frequency part of the feature containing specific information. This means that there is a game relationship between anti-aliasing and preserving high-frequency information. The present invention adjusts the influence between the two by dynamically learning the low-pass filter cutoff frequency to obtain the optimal solution.

[0021] like Figure 1 As shown, the low-pass filtering method for time axis positioning of an action of the present invention comprises the following steps:

[0022] Step 1: Use a spatiotemporal feature extractor (such as TSN, etc.) to extract features of the entire input video;

[0023] Step 2: Based on the extracted features of the entire input video, the action proposal is calculated using the relevant action proposal algorithm;

[0024] Step 3: Use a low-pass filter to filter out some high-frequency information for the features corresponding to each action proposal, that is, the high-frequency passband of the regional features corresponding to the action proposal;

[0025] Step 4: Use temporal downsampling to calculate the features corresponding to the action proposal; and

[0026] Step 5: Use the features corresponding to the action proposal to calculate the final position and category of the action. Specifically, the features corresponding to the action proposal in step 4 are input into the classifier and regressor to calculate the final position and category of the action, and output the final predicted action result.

[0027] Different from the traditional action timing detection algorithm, the present invention introduces a low-pass filter before timing downsampling to suppress the high-frequency passband. At the same time, in order to balance the impact between anti-aliasing and preserving high-frequency information, a neural network is used to dynamically learn different cutoff frequencies for each action. In this way, for actions that change slowly, the neural network can learn a lower cutoff frequency, thereby discarding more high-frequency parts in exchange for higher anti-aliasing benefits. For actions that change dramatically, the neural network can learn a higher cutoff, thereby preserving more effective information for subsequent classification and regression. This dynamic game between anti-aliasing and preserving high-frequency information can achieve better detection performance.

[0028] The present invention conducted experiments on the widely used THUMOS'14 dataset, using mean average precision (mAP) as the evaluation index, and the results are shown in Table 1:

[0029] Table 1 Test results of low-pass filtering anti-aliasing on THUMOS'14 dataset (mAP@0.5)

[0030]

[0031] Among them, the low-pass filtering + average pooling of the present invention achieved the best performance. It surpassed the original average pooling 8.02% mAP@0.5, which proves that the use of low-pass filters to remove aliasing problems proposed by the present invention can effectively improve the performance of the action detection model. In addition, as shown in Table 1 above, on the THUMOS'14 dataset, the present invention also far surpassed other traditional temporal downsampling methods (such as average pooling, maximum pooling, attention mechanism pooling and Gaussian pooling in Table 1), which proves the advancement of the present invention. The method used in the present invention has a wide range of applicability and can be integrated into most existing temporal detection models of actions.

[0032] The present invention also conducts experiments on the widely used ActivityNet 1.3 dataset, using the area under the curve (AUC) as the evaluation indicator. As shown in Table 2, using low-pass filtering to remove aliasing on the ActivityNet dataset also improves the model performance compared to average sampling.

[0033] Table 2 Test results (AUC) of low-pass filtering anti-aliasing on ActivityNet 1.3 dataset

[0034]

[0035] In addition, we visualized the heat map of the aliasing problem in the ActivityNet dataset with and without low-pass filtering in the frequency domain, such as Figure 2a and 2b As shown in the figure, the horizontal axis represents different frequencies, while the vertical axis represents channels with different characteristics. Dark colors represent large differences, while light colors represent small differences. Traditional time series downsampling often leads to serious aliasing problems ( Figure 2a ), which means that there is a significant difference in the information contained in the features before and after sampling. After using the low-pass filter, the aliasing error in the low-frequency passband is greatly suppressed ( Figure 2b ). It is worth noting that the low-pass filter selects a compromise sampling frequency, which is chosen to retain some high-frequency information at the expense of tolerating some aliasing errors.

[0036] It should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

Claims

1. A low-pass filtering method for temporal positioning of an action, characterized in that: The steps include: Step 1: Use the spatiotemporal feature extractor to extract features of the entire input video; Step 2: According to the extracted features of the entire input video, an action proposal is calculated using a relevant action proposal algorithm; Step 3: Use a low-pass filter to filter out part of the high-frequency information of the feature corresponding to each action proposal, where the part of the high-frequency information is the high-frequency passband of the regional feature corresponding to the action proposal; Step 4: Use temporal downsampling to calculate the features corresponding to the action proposal; Step 5: Input the features corresponding to the action proposal into the classifier and regressor, calculate the final position and category of the action, and output the final predicted action result; wherein, in step 3, a neural network is used to dynamically learn different truncation frequencies for each action; for actions that change slowly, the neural network learns a lower truncation frequency; for actions that change dramatically, the neural network learns a higher truncation frequency.

Images