Feature deduplication time action positioning method based on highlighted frame retention strategy

Abstract

The invention discloses a feature deduplication time action positioning method based on a highlighted frame retention strategy, which comprises the following steps of: performing block processing on a video stream, and converting the video stream into a feature stream; constructing a multi-time-domain feature pyramid; constructing a prominent frame feature pyramid; constructing a redundancy elimination pyramid model; and training the feature redundancy elimination pyramid model. The multi-time-domain sensing module designed by the invention can fuse local feature dependency information and global feature dependency information and pay attention to actions with long duration and actions with short duration at the same time, so that the problem of potential feature loss is relieved. The feature redundancy elimination pyramid model constructed by the invention performs feature retention and feature enhancement on highlighted frame features, and better positioning and classification precision is obtained in a video positioning task.

Description

Technical Field

[0001] The present invention belongs to the field of image data processing technology, and further relates to a feature-redundant temporal action positioning method based on a prominent frame retention strategy in the field of video understanding technology. The present invention can be applied to time point segmentation in scenes such as sports events, automatic monitoring alarms, and movie clip processing. Background Art

[0002] Compared to images, videos contain rich and continuous action information in the temporal dimension. However, in a short time span, the action usually has only slight changes, which leads to extremely high similarity of video frames in a short time range, and the corresponding video features are highly redundant in the local time. In the task of temporal action localization, it is necessary not only to complete the action recognition task, but also to accurately locate the start and end time of the action. These redundant frame features will cause the real action start and end frames to have highly similar features to the adjacent frames, causing the model to mistakenly use the frame where the adjacent action starts as the predicted start frame, which in turn causes the positioning time to shift. The rapid development of Internet videos has led to an increasing demand for automated processing of uncut videos.

[0003] The underlying implementations of the current mainstream methods mainly include convolution-based design architectures and Transformer-based design architectures. The upper-level mainstream designs are mainly divided into two-stage temporal action localization networks and single-stage temporal action localization networks. The proposal generation methods of the model are divided into anchor-based localization methods and anchor-free localization methods.

[0004] Stehouwer J et al. proposed a temporal action localization method without anchor points based on Transformer architecture in their paper “Actionformer: Localizing moments of actions with transformers” (European Conference on Computer Vision (ECCV), pp492-510, 2022). The implementation steps of this method are: after preprocessing the video, use Transformer to build an attention relationship graph between frames, and combine the local multi-head attention mechanism to improve the richness of the relationship. In order to adapt to action information of different time spans, the combined time pyramid structure adapts to more actions of different durations according to different spans. The overall positioning ability of temporal action localization is improved, and it is effectively proved that the attention mechanism can improve the action localization ability in temporal action localization. However, the shortcomings of this method are that when constructing the pyramid, only downsampling is used to change the time span. When there is highly redundant information in the video height, it is easy to lose the key frames used to distinguish the start and end of the action. The key action information is lost in the process of constructing the pyramid, which leads to positioning offset, resulting in the impact of recognition accuracy in the process of temporal action localization.

[0005] Hangzhou Dianzi University disclosed a method for temporal action localization based on a pyramid structure using a graph attention network in its patent application "A method for temporal action localization based on a pyramid structure using a graph attention network" (application number: 202110412867.8, application publication number: CN 113255443B). It mainly involves the use of a multi-scale pyramid model built by a graph attention network to fuse and predict behavior detection with multiple time spans. First, the data is preprocessed to extract the initial spatiotemporal features of the video data. Next, a multi-scale model with a temporal attention mechanism is built based on the pyramid network model of the graph attention mechanism. Finally, the positioning detection result is generated. The graph attention mechanism is introduced, and an attention module is added to the temporal information, which fully integrates the temporal content in the features. However, the method still has the following shortcomings: first, the long-range dependency extraction ability of the graph attention mechanism is not as good as that of the Transformer architecture, and the problem of prominent frame information loss caused by the construction of the feature pyramid is ignored. In addition, no feature extraction is performed for local action correlation information, which will lose the detailed information of the video frame, resulting in the extraction of insufficient features, which will affect the final recognition accuracy. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to provide a feature-redundant temporal action localization method based on a salient frame retention strategy, so as to solve the problems of ignoring feature redundancy and loss of salient frame features in the process of constructing traditional feature pyramids, resulting in a decrease in the accuracy of positioning time nodes during the positioning process, and only using a single feature extraction strategy, resulting in low feature richness during feature extraction, affecting the accuracy of temporal positioning.

[0007] The technical idea for achieving the purpose of the present invention is that the present invention constructs a feature redundancy elimination pyramid model, retains the prominent frame features in the pre-extracted video features, and finally merges them with the multi-time domain feature pyramid. Among them, the multi-time domain feature pyramid performs more fine-grained feature extraction on the video features, and the prominent frame feature pyramid is to alleviate the problem of increased redundancy caused by the multi-time domain feature pyramid during the construction process. It solves the problem in the prior art that when constructing the feature pyramid, only the inter-layer extraction module is used to cause the loss of prominent frame features and the increased feature redundancy, resulting in reduced accuracy in the positioning process. At the same time, the present invention designs a multi-time domain perception module, which is composed of a multi-head attention module and a large-core convolution module in parallel. The large-core convolution module captures the local feature dependency information in the feature stream through the convolution receptive field, and the multi-head attention module captures the global feature dependency information in the feature stream. In order to obtain richer action features, it can pay attention to actions with long duration and actions with short duration at the same time, solving the problem that it is difficult to obtain rich features with a single feature extraction, potential information loss, and ultimately poor model accuracy.

[0008] The steps of the positioning method of the present invention include the following:

[0009] Step 1, after dividing the input original video stream into blocks, the image video stream is converted into a feature stream;

[0010] Step 2: construct a multi-time domain feature pyramid to extract local information and fuse it with global information;

[0011] Step 3, adopting the prominent frame retention strategy, after performing one-dimensional convolution on the original feature stream to map the prominent features, only downsampling operation in the time domain is performed according to the time span feature, and the prominent frame features in the original video are extracted through the prominent frame feature pyramid;

[0012] Step 4, constructing a redundant pyramid model consisting of a multi-time domain feature pyramid and a salient frame feature pyramid in parallel;

[0013] Step 5, training the feature redundancy elimination pyramid model;

[0014] Step 6: In the same manner as step 1, the feature stream of the video stream to be positioned is obtained, the feature stream is input into the trained redundant pyramid model, and the positioning result is output.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] First, since the present invention constructs a feature redundancy elimination pyramid model, in which a multi-time domain feature pyramid extracts local information and fuses it with global information, and a salient frame feature pyramid is used to retain salient frame features that play a key role in the positioning task, the problem of salient frame feature loss and increased feature redundancy caused by only using an inter-layer extraction module when constructing a feature pyramid in the prior art is avoided. This allows the present invention to retain and enhance the features of the salient frame features, thereby obtaining better positioning and classification accuracy in the video positioning task.

[0017] Second, the present invention designs a multi-time domain perception module, which integrates local feature dependency information and global feature dependency information to obtain richer action features, so that the present invention can overcome the shortcomings of the prior art that only a single feature extraction is difficult to obtain rich features and potential information loss. The present invention can focus on actions with long duration and actions with short duration at the same time, thereby alleviating the problem of potential feature loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention; DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0020] Reference Figure 1 , the implementation steps of the embodiment of the present invention are further described in detail.

[0021] Step 1: Divide the original video stream into blocks.

[0022] The input video data is processed into blocks of continuous frames. The sliding window segmentation method is used. Each window extracts a group of continuous frames. The window slides through the entire video with a preset time span to obtain a collection of multiple groups of continuous frames of the current input video. Among them, x k represents the kth group of continuous frames, and K represents the total number of continuous frame blocks of the current input video.

[0023] The embodiment of the present invention uses the Thumos14 video dataset. For each input video in the Thumos14 dataset, the image frames are first extracted at 30FPS, and the sliding window is used for block division. The sliding window size is 16 and the window movement span is 4. The dataset is divided into blocks to obtain a set of multiple groups of continuous frames of the current input video.

[0024] Step 2: Convert the image stream into feature stream.

[0025] Each group of continuous frames is input into the feature extraction model I3D in turn to extract the optical flow features and image features of the image video stream, and the two extracted features are horizontally spliced ​​to obtain the feature stream of the group. Among them, f k ∈R 256*2048 represents the feature stream of the kth group, whose feature dimension is 2048 and time dimension is 256. The image features representing the kth group of consecutive frames have a feature dimension of 1024 and a time dimension of 256. The optical flow features representing the kth group of consecutive frames have a feature dimension of 1024 and a time dimension of 256.

[0026] Step 3: construct a multi-time domain feature pyramid to fuse the extracted local information with the global information.

[0027] The multi-time domain feature pyramid is composed of a multi-head attention module and a large kernel convolution module in parallel. The large kernel convolution module captures the local feature dependency information in the feature stream through the convolution receptive field, and the multi-head attention module captures the global feature dependency information in the feature stream. The local feature dependency information and the global feature dependency information are added together as the output of the multi-time domain feature pyramid.

[0028] The large-core convolution module contains two feature extraction branches, upper and lower. The upper branch is a feature enhancement module, which consists of an average pooling layer with a convolution kernel size of 3 and a step size of 1, a fully connected layer consisting of a one-dimensional convolution with a convolution kernel size of 1 and 256 convolutions, and is connected in series by the activation function RELU. The lower branch is a feature extraction module, which is a one-dimensional convolution with a convolution kernel size of 1 and a large-core convolution with a convolution kernel size of 9 in parallel and then multiplied by a one-dimensional convolution with a convolution kernel size of 1. The upper and lower branches are used as the output dimension of the large-core convolution module through feature addition, and the dimension is 256*2048. In order to construct a pyramid structure, downsampling MaxPool is also used to obtain 5 scales with different time spans T={256,128,64,32,16}.

[0029] The lower branch is a multi-head attention module, where the input features and learnable parameters {W Q ,W K ,W V} respectively, where the learnable parameter dimension is 256*256, which is the same as {W Q ,W K The calculated result is passed through the softmax layer to obtain the feature similarity matrix, the matrix dimension is 256*256, and {W VThe result of the multiplication is then multiplied by a normalization layer LN (dropout = 0.01), a multi-head attention layer MSA, a normalization layer LN and a multi-layer perceptron MLP in series to obtain the global feature dependency information of the input feature stream. k The dimension is 256*2048. In order to construct the pyramid structure, MaxPool is used to obtain 5 scales with different time spans T={256, 128, 64, 32, 16}.

[0030] Global feature dependency information Dependence information on local features After being fused by feature addition, it is output as a multi-time domain feature pyramid:

[0031] F a =Global+Local,

[0032] Among them, F i a It is the output of the multi-time domain feature pyramid.

[0033] The specific implementation steps are as follows:

[0034] Step 1: Feature Flow Obtain local feature dependency information through the large kernel convolution module k :

[0035]

[0036]

[0037] Among them, FC represents the fully connected layer mapping operation, which is implemented by one-dimensional convolution, Conv k It represents the operation of extracting features using large kernel convolution, the convolution kernel size is 9, ReLU represents the nonlinear mapping of the input, AvgPool represents the average pooling operation, the size of the pooling window is 3*3, and the step size is 1.

[0038] Step 2: Feature Flow After the multi-head attention module, the input features and the learnable parameters {W Q ,W K ,W V} respectively, where {W Q ,W K The result after calculation is passed through the softmax layer to obtain the feature similarity, which is consistent with {W V The result of the multiplication is then multiplied by the normalization layer LN, the multi-head attention layer MSA, the normalization layer LN and the multi-layer perceptron MLP in series to obtain the global feature dependency information of the input feature stream.k as follows:

[0039]

[0040]

[0041]

[0042] Among them, MLP represents a multi-layer perceptron, which consists of multiple nonlinear mapping layers, LN represents layer normalization operation, MSA represents the use of multi-head attention mechanism to extract comprehensive features, and W Q ,W K ,W V Indicates the learnable parameters Query, Key, Value, is the scaling factor, and the superscript T indicates the transpose operation.

[0043] The third step is to integrate the global feature dependency information Dependence information on local features The output F of the multi-time domain feature pyramid is fused by feature addition. i a :

[0044] F a =Global+Local,

[0045] Among them, Global represents the global feature dependency information of the video extracted by the multi-head attention module, and Local represents the local feature dependency information of the video extracted by the large-kernel convolution module.

[0046] Step 4: Construct a salient frame feature pyramid.

[0047] The input feature stream The original features are mapped to highlight information. The mapping is implemented by a one-dimensional convolution layer with a kernel size of 1, a span of 1, and a number of convolution kernels of 256. The features are downsampled in the feature time dimension, and MaxPool is used to obtain 5 scales with different time spans T = {256, 128, 64, 32, 16}. The salient frame features of the input feature stream at each time span are obtained. The obtained results are horizontally spliced ​​to obtain

[0048] The salient frame feature pyramid will input the feature flow The original features are mapped to highlight information. The mapping is implemented by a one-dimensional convolutional layer with a kernel size of 1 and a span of 1. The features are downsampled in the feature time dimension to obtain the salient frame features of the input feature stream at each time span. The obtained results are horizontally spliced ​​to obtain

[0049]

[0050] Where layer represents the number of pyramid layers. When layer=1, it means that this is the first layer of the pyramid and the original features need to be mapped to the prominent frame feature. i Represents the input of the i-th layer of the pyramid, and MaxPool represents the maximum pooling operation.

[0051] Step 5: construct a redundant pyramid model consisting of a multi-time domain feature pyramid and a salient frame feature pyramid in parallel.

[0052] The feature redundancy elimination pyramid model is to highlight the feature output F of the frame feature pyramid b The feature output F of the multi-time domain feature pyramid a , after feature enhancement of the salient frame through feature addition operation, the output F of the feature redundancy elimination pyramid model is obtained:

[0053] F=α*F a +β*F b ,

[0054] Among them, α represents the fusion factor of the multi-time domain perception module pyramid, and β represents the fusion factor of the salient frame retention strategy pyramid. The introduction of the fusion factor is to give the salient frame a larger feature weight to improve the effect of feature highlighting.

[0055] Step 6: Train the feature redundancy elimination pyramid model.

[0056] The multi-layer features obtained by the salient frame feature pyramid model are put into two branch structures for each layer. The first branch is for category prediction, which predicts the action category of the current frame and compares it with the actual action label in the video segment, and uses focal loss FocalLoss to calculate the classification loss; the second branch is used to predict the possible start and end moments of the action, and uses smooth L1 loss to calculate the positioning loss. The results of the two loss functions are summed and divided by the number of positive samples as the training loss. The gradient back-propagation algorithm is used to iteratively update the model parameters until the network model loss function converges to obtain a trained feature redundancy-eliminating pyramid model.

[0057] The network model loss function is as follows: the generated positioning information is compared with the actual positioning label in the video segment, and the smooth L1 loss is used to calculate the positioning loss. The results calculated by the two loss functions are summed and divided by the number of positive samples as the training loss:

[0058]

[0059]

[0060]

[0061] Among them, L cls Represents the difference between the class label predicted by the model and the true label, implemented by the focal loss, L reg It represents the difference between the time boundary predicted by the model and the actual time boundary, which is achieved by L1 smoothing loss, λ reg represents the weight hyperparameter, which can control how much the model focuses on the regression loss.

[0062] Step 7: Generate positioning results.

[0063] The same method as step 1 and step 2 is used to obtain the feature stream of the video stream to be located, and the feature stream is input into the trained redundant pyramid model for output block processing, and then the feature stream is converted to obtain the feature stream F i Represents the i-th video block. An entire video will be divided into multiple video blocks. These video blocks will be horizontally spliced ​​and combined into F. This feature F will be input into the classification head and the localization head to obtain the action category and grid offset. Among them, the classification head is realized by a one-dimensional convolutional layer with a kernel size of 3 and a kernel number of 256, and an average pooling layer with a kernel size of 3 and a span of 3 in series.

[0064] The localization head is implemented by a one-dimensional convolutional layer with a kernel size of 1 and a kernel number of 256:

[0065] c t =ArgMax{Conv1D(F t )},

[0066] d st =Conv1D(F t ),

[0067] d et rConv1D(F t ),

[0068] Among them, d st represents the grid offset between the start time of the action at time t and t, d et Indicates the grid offset between the action end time at time t and t. The grid offset is in percentage format and needs to be mapped to the real time. Use mathematical calculations to map the predicted grid time to the actual time and combine the output Y t ={s t ,e t ,c t}:

[0069]

[0070]

[0071] Among them, t represents the number of layers of the current feature pyramid, t = {256, 128, 64, 32, 16}, a total of 5 scales, 2 k-1 Indicates the time step of the current layer (layer scaling ratio 2:1), represents the start time of the action at time t in the Kth time span, It represents the end time of the action at time t in the Kth time span, and finally non-maximum suppression NMS is used to obtain the optimal result.

[0072] The effect of the present invention can be further illustrated by the following simulation experiment.

[0073] 1. Simulation conditions.

[0074] The simulation experiment of the present invention adopts the software Pycharm as a simulation tool, and the computer configuration is as follows: motherboard: Gigabyte Z490 AORUS PRO AX, CPU: Intel (R) Core (TM) i7-10700KF CPU @ 3.80 GHz, memory: 64 GB, graphics card: Gigabyte GAMingOC GeForce RTX 3090*2, system: Ubuntu 24.04.

[0075] The simulation experiments of the present invention use data from Thumos14 and MultiThumos14 video datasets, which were created by a research team at Stanford University and are mainly used for video understanding and computer vision research in academia and industry. In the Thumos14 dataset, each segment is labeled with only a single action category. MultiThumos14 expands the original Thumos14 dataset and adds multi-action annotations, that is, a video can have multiple different action labels at a certain moment.

[0076] 2. Analysis of simulation experiment content and effects.

[0077] There are three simulation experiments of the present invention.

[0078] Simulation 1: The present invention and three existing technologies (ActionFormer algorithm, TemporalMaxer algorithm, TriDet algorithm) are used to perform prediction simulation on Thumos14 data. Using the tIoU evaluation index, performance evaluation is performed at mAP@[0.3:0.1:0.7], and the performance evaluation results of the four methods are shown in Table 1.

[0079] The three prior arts refer to:

[0080] The ActionFormer algorithm is an algorithm proposed by Stehouwer J et al. in their paper "Actionformer: Localizing moments of actions with transformers" (European Conference on Computer Vision (ECCV), pp 492-510, 2022), referred to as the Actionformer algorithm.

[0081] The TemporalMaxer algorithm is an algorithm proposed by Tang TN et al. in their paper “Temporalmaxer: Maximizetemporal context with only maxpooling for temporal action localization” (arXivpreprint arXiv, pp 202, 2022), referred to as the TemporalMaxer algorithm.

[0082] The TriDet algorithm is an algorithm proposed by Shi D et al. in their paper "Tridet: Temporal action detection with relative boundary modeling" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. pp18857-18866, 2023), referred to as the TriDet algorithm.

[0083] Table 1 Thumos14 data simulation results evaluation list

[0084] mAP The present invention ActionFormer TemporalMaxer TriDet 0.3 84.3 82.1 82.8 83.6 0.4 80.5 77.8 78.9 80.1 0.5 73.8 71.0 71.8 72.9 0.6 62.2 59.4 60.5 62.4 0.7 48.0 43.9 44.7 47.4 Avg 69.7 66.8 67.7 69.3

[0085] Simulation 2: The present invention and three existing technologies (ActionFormer algorithm, TemporalMaxer algorithm, TriDet algorithm) are used to perform prediction simulation on MultiTHUMOS14 data respectively. The tIoU evaluation index is used to perform performance evaluation at mAP@[0.2:0.2:0.7]. The results are shown in Table 2.

[0086] Table 2 MultiTHUMOS14 data simulation results evaluation table

[0087] mAP The present invention ActionFormer TemporalMaxer TriDet 0.2 49.6 46.4 47.5 49.1 0.5 35.0 32.4 33.4 34.3 0.7 19.8 15.0 17.4 17.8 Avg 31.2 28.6 29.9 30.7

[0088] As can be seen from Table 2, in the AvgmAP evaluation index used in the experiment, the present invention is higher than the prior art, which proves that compared with the prior art, the present invention can improve the result of temporal action localization, and MultiTHUMOS14 is an expanded version of the Thumos14 dataset, with finer categories and more action localization scenarios, which proves that the present invention can outperform the existing methods in more difficult tasks.

[0089] The above results show that the present invention can better locate the time points when actions start and end in uncut videos. The feature pyramid architecture of the salient frames is retained to reduce the influence of feature similarity caused by inter-layer operations on positioning. Through the multi-time domain perception module, we fuse global and local temporal features to perform more comprehensive feature extraction. Through experiments, it is proved that the present invention has achieved significant results on most data sets.

Claims

1. A feature-redundant temporal action localization method based on a prominent frame retention strategy, characterized in that: A multi-time domain feature pyramid is constructed for extracting the fusion of local information and global information, and a prominent frame retention strategy is adopted to extract the prominent frame features in the original video through the prominent frame feature pyramid; the steps of the positioning method include the following: Step 1, after dividing the input original video stream into blocks, the image video stream is converted into a feature stream; Step 2: construct a multi-time domain feature pyramid to extract local information and fuse it with global information; Step 3, adopting the prominent frame retention strategy, after performing one-dimensional convolution on the original feature stream to map the prominent features, only downsampling operation in the time domain is performed according to the time span feature, and the prominent frame features in the original video are extracted through the prominent frame feature pyramid; Step 4, constructing a redundant pyramid model consisting of a multi-time domain feature pyramid and a salient frame feature pyramid in parallel; Step 5, training the feature redundancy elimination pyramid model; Step 6: In the same manner as step 1, the feature stream of the video stream to be positioned is obtained, the feature stream is input into the trained redundant pyramid model, and the positioning result is output.

2. The feature-redundant time action localization method based on the prominent frame retention strategy according to claim 1 is characterized in that: The original video stream segmentation in step 1 refers to the segmentation of the input video data into continuous frames, using a sliding window segmentation method, extracting a group of continuous frames from each window, and sliding the window through the entire video with a preset time span to obtain a set of multiple groups of continuous frames of the current input video. Among them, x k represents the kth group of continuous frames, and K represents the total number of continuous frame blocks of the current input video.

3. The method for locating action in feature-redundant time based on prominent frame retention strategy according to claim 1, characterized in that: The conversion of the image video stream into a feature stream in step 1 refers to inputting each group of continuous frames into the feature extraction model I3D in sequence, extracting the optical flow features and image features of the image video stream, and horizontally splicing the two extracted features to obtain the feature stream of the group. Among them, f k represents the feature flow of the kth group, represents the image features of the kth group of consecutive frames, Represents the optical flow features of the kth group of consecutive frames.

4. The feature-redundant time action localization method based on the prominent frame retention strategy according to claim 1 is characterized in that: The multi-time domain feature pyramid described in step 2 is composed of a multi-head attention module and a large kernel convolution module in parallel; the large kernel convolution module captures the local feature dependency information in the feature stream through the convolution receptive field, and the multi-head attention module captures the global feature dependency information in the feature stream. The local feature dependency information and the global feature dependency information are added as the output of the multi-time domain feature pyramid; the large kernel convolution module contains upper and lower feature extraction branches, the upper branch is a feature enhancement module, which is composed of an average pooling layer, a fully connected layer, and an activation function in series; the lower branch is a feature extraction module, which is a one-dimensional convolution and a large kernel convolution in parallel and then multiplied by a one-dimensional convolution, and the upper and lower branches are output by adding features as the large kernel convolution module; the specific implementation steps are as follows: Step 1: Feature Flow Obtain local feature dependency information through the large kernel convolution module k : Among them, FC represents the fully connected layer mapping operation, which is implemented by one-dimensional convolution, Conv k Represents the operation of extracting features using large kernel convolution, the convolution kernel size is 9, ReLU represents nonlinear mapping of the input, AvgPool represents the average pooling operation, the pooling window size is 3*3, and the step size is 1; Step 2: Feature Flow After the multi-head attention module, the input features and the learnable parameters {W Q ,W K, W V } respectively, and {W Q ,W K The result after calculation is passed through the softmax layer to obtain the feature similarity, which is consistent with {W V The result of the multiplication is then multiplied by the normalization layer LN, the multi-head attention layer MSA, the normalization layer LN and the multi-layer perceptron MLP in series to obtain the global feature dependency information of the input feature stream. k : Among them, MLP represents a multi-layer perceptron, which consists of multiple nonlinear mapping layers, LN represents layer normalization operation, MSA represents the use of multi-head attention mechanism to extract comprehensive features, and W Q ,W K ,W V Indicates the learnable parameters Query, Key, Value, is the scaling factor, and the superscript T indicates the transposition operation; The third step is to integrate the global feature dependency information Dependence information on local features The output F of the multi-time domain feature pyramid is fused by feature addition. i a : F a =Global+Local, Among them, Global represents the global feature dependency information of the video extracted by the multi-head attention module, and Local represents the local feature dependency information of the video extracted by the large-kernel convolution module.

5. The feature-redundant time action localization method based on the prominent frame retention strategy according to claim 1 is characterized in that: The salient frame feature pyramid described in step 3 converts the input feature stream The original features are mapped to highlight information. The mapping is implemented by a one-dimensional convolutional layer with a kernel size of 1 and a span of 1. The features are downsampled in the feature time dimension to obtain the salient frame features of the input feature stream at each time span. The obtained results are horizontally spliced ​​to obtain Where layer represents the number of pyramid layers. When layer=1, it means that this is the first layer of the pyramid and the original features need to be mapped to the prominent frame feature. i Represents the input of the i-th layer of the pyramid, and MaxPool represents the maximum pooling operation.

6. The feature-redundant time action localization method based on the prominent frame retention strategy according to claim 1 is characterized in that: The feature redundancy elimination pyramid model described in step 4 is to transform the feature output F of the salient frame feature pyramid b The feature output F of the multi-time domain feature pyramid a , through the feature addition operation, after the feature enhancement of the salient frame, the output F of the feature redundancy elimination pyramid model is obtained: F=α*F a +β*F b , Among them, a represents the fusion factor of the pyramid of multi-temporal perception modules, and β represents the fusion factor of the pyramid of salient frame retention strategy.

7. The method for locating action in feature-redundant time based on prominent frame retention strategy according to claim 1, characterized in that: The steps of training the redundant pyramid model described in step 5 are as follows: the multi-layer features obtained by the salient frame feature pyramid model, each layer of features are placed into two branch structures, the first branch is the category prediction, predicts the action category of the current frame, compares it with the actual action label in the video segment, and uses the focal loss to calculate the classification loss; the second branch is used to predict the possible start and end moments of the action; the gradient back propagation algorithm is used to iteratively update the model parameters until the loss function of the entire network model converges, and a trained redundant pyramid model is obtained.

8. The method for locating action in feature-redundant time based on prominent frame retention strategy according to claim 7, characterized in that: The network model loss function is as follows: Among them, L cls Represents the difference between the class label predicted by the model and the true label, which is implemented by the focal loss, L reg It represents the difference between the time boundary predicted by the model and the actual time boundary, which is achieved by L1 smoothing loss, λ reg represents the weight hyperparameter.

9. The method for locating action in feature-redundant time based on prominent frame retention strategy according to claim 1, characterized in that: The steps for generating the positioning result in step 6 are as follows: input the feature stream into the trained redundant pyramid model, output block processing, and then transform the feature stream to obtain the feature stream F i Represents the i-th video block. An entire video will be divided into multiple video blocks. These video blocks are horizontally spliced ​​and combined into F. This feature F will be input into the classification head and the localization head to obtain the action category and grid offset. Among them, the classification head is implemented by a one-dimensional convolution layer with a kernel size of 3 and an average pooling layer with a kernel size of 3 in series; the localization head is implemented by a one-dimensional convolution layer with a kernel size of 1; the specific implementation formula is as follows: c t =ArgMax{Conv1D(F t )}, d st =Conv1D(F t ), d et =Conv1D(F t ), Among them, d st represents the grid offset between the start time of the action at time t and t, d et Indicates the grid offset between the action end time at time t and t. The grid offset is in percentage format and needs to be mapped to the real time. Use mathematical calculations to map the predicted grid time to the actual time and combine them to output Y t ={s t ,e t ,c t }; t represents the number of layers of the current feature pyramid, 2 k-1 Indicates the time step of the current layer, the scaling ratio of the layer is 2:1, represents the start time of the action at time t in the Kth time span, It represents the end time of the action at time t in the Kth time span, and finally non-maximum suppression NMS is used to obtain the optimal result.

Images