Feature redundancy-elimination time action positioning method based on highlight frame reservation strategy

Abstract

This invention discloses a feature deduplication temporal action localization method based on a prominent frame preservation strategy. The steps are as follows: video stream segmentation and conversion into a feature stream; construction of a multi-temporal-domain feature pyramid; construction of a prominent frame feature pyramid; construction of a deduplication pyramid model; and training of the feature deduplication pyramid model. The multi-temporal-domain perception module designed in this invention can fuse local and global feature dependency information, simultaneously focusing on both long-duration and short-duration actions, mitigating the problem of potential feature loss. The feature deduplication pyramid model constructed in this invention preserves and enhances prominent frame features, achieving better localization and classification accuracy in video localization tasks.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, and further relates to a feature deduplication and temporal action localization method based on a prominent frame preservation strategy in the field of video understanding technology. This invention can be applied to time point segmentation in scenarios such as sports events, automatic alarm monitoring, and film clip processing. Background Technology

[0002] Compared to images, videos contain rich and continuous action information over time. However, within a short time span, actions typically exhibit only minor changes, leading to extremely high similarity among video frames within a short time range. Consequently, video features exhibit significant redundancy in local timeframes. In temporal action localization tasks, it is necessary not only to recognize actions but also to accurately pinpoint the start and end times of the actions. These redundant frame features cause the true start and end frames of the action to have highly similar characteristics to adjacent frames, leading the model to incorrectly identify the frame where the action begins as the predicted start frame, thus causing a shift in localization time. The rapid development of internet video has led to a continuously increasing demand for automated processing of unedited video.

[0003] The current mainstream methods primarily employ convolutional architectures and Transformer architectures at the underlying level. Upper-level mainstream designs mainly fall into two-stage temporal action localization networks and single-stage temporal action localization networks. Proposal generation methods for the models are categorized into anchor-based localization methods and anchorless localization methods.

[0004] Stehouwer J et al., in their paper "Actionformer: Localizing moments of actions with transformers" (European Conference on Computer Vision (ECCV), pp492-510, 2022), proposed an anchorless temporal action localization method based on the Transformer architecture. The method involves preprocessing the video, constructing an attention graph between frames using Transformer, and combining this with a local multi-head attention mechanism to enhance the richness of these relationships. To accommodate action information across different time spans, a temporal pyramid structure is incorporated to accommodate actions of varying durations based on the time span. This improves the overall localization capability of temporal action localization and effectively demonstrates that the attention mechanism can enhance action localization in temporal action localization. However, this method still has shortcomings. When constructing the pyramid, it only uses downsampling to change the time span. Given the high redundancy in video height, key frames used to distinguish the start and end of actions are easily lost. This loss of key action information during pyramid construction leads to localization offset, affecting the accuracy of temporal action localization.

[0005] Hangzhou Dianzi University disclosed a temporal action localization method based on a graph attention network with a pyramid structure in its patent application "A Temporal Action Localization Method Based on a Pyramid Structure Graph Attention Network" (Application No.: 202110412867.8, Publication No.: CN 113255443B). This method mainly involves using a multi-scale pyramid model built with a graph attention network to fuse and predict actions across multiple time spans. First, the data is preprocessed to extract the initial spatiotemporal features of the video data. Next, a multi-scale model with a temporal attention mechanism is built based on a pyramid network model with a graph attention mechanism. Finally, the localization detection result is generated. The graph attention mechanism is introduced, adding an attention module to the temporal information and fully integrating temporal content into the features. However, this method still has shortcomings. First, the long-range dependency extraction capability of the graph attention mechanism is not as good as that of the Transformer architecture. It also ignores the problem of lost salient frame information caused by the construction of the feature pyramid. Furthermore, it does not extract features for local action correlation information, resulting in the loss of detailed information of video frames and insufficient feature extraction. All of these factors affect the final recognition accuracy. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technologies by providing a feature redundancy reduction time action localization method based on a prominent frame retention strategy. This method solves the problems of neglecting feature redundancy and prominent frame feature loss during the construction of traditional feature pyramids, which leads to a decrease in the accuracy of localization time nodes. It also addresses the issue of low feature richness during feature extraction due to the use of only a single feature extraction strategy, which affects the accuracy of time localization.

[0007] The technical approach to achieving the objectives of this invention is as follows: This invention constructs a feature redundancy-reduction pyramid model, which retains prominent frame features from pre-extracted video features and ultimately fuses them with a multi-temporal feature pyramid. The multi-temporal feature pyramid performs finer-grained feature extraction from the video features, while the prominent frame feature pyramid mitigates the redundancy exacerbation problem caused during its construction. This solves the problem in existing technologies where the use of only inter-layer extraction modules during feature pyramid construction leads to the loss of prominent frame features and increased feature redundancy, resulting in reduced accuracy during localization. Simultaneously, this invention designs a multi-temporal perception module, composed of a multi-head attention module and a large-kernel convolution module connected in parallel. The large-kernel convolution module captures local feature dependency information in the feature stream through the convolutional receptive field, while the multi-head attention module captures global feature dependency information in the feature stream. To obtain richer action features, it can simultaneously focus on both long-duration and short-duration actions, solving the problem that single feature extraction is insufficient to obtain rich features, potentially leading to information loss and ultimately poor model accuracy.

[0008] The positioning method of the present invention includes the following steps:

[0009] Step 1: Divide the input raw video stream into blocks, and then convert the image video stream into a feature stream;

[0010] Step 2: Construct a multi-time-domain feature pyramid, extract local information and fuse it with global information;

[0011] Step 3: Emphasis frame preservation strategy is adopted. After performing one-dimensional convolution on the original feature stream to map the emphasis features, downsampling operation is performed only on the time span features. Emphasis frame features in the original video are extracted through the emphasis frame feature pyramid.

[0012] Step 4: Construct a redundancy-eliminating pyramid model consisting of a multi-time-domain feature pyramid and a prominent frame feature pyramid connected in parallel;

[0013] Step 5: Train the feature redundancy-reduction pyramid model;

[0014] Step 6: Using the same method as in Step 1, obtain the feature stream of the video stream to be located, input the feature stream into the trained redundancy-reducing pyramid model, and output the localization result.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] First, because this invention constructs a feature redundancy reduction pyramid model, in which a multi-temporal feature pyramid extracts and fuses local and global information, and a prominent frame feature pyramid is used to retain prominent frame features that play a key role in the localization task, it avoids the problems of prominent frame feature loss and increased feature redundancy caused by using only inter-layer extraction modules when constructing feature pyramids in the prior art. This allows the present invention to obtain better localization and classification accuracy in video localization tasks by preserving and enhancing prominent frame features.

[0017] Second, the present invention designs a multi-temporal domain perception module that integrates local feature-dependent information and global feature-dependent information to obtain richer action features. This allows the present invention to overcome the shortcomings of existing technologies that use only a single feature extraction method to obtain rich features and the potential loss of information. The present invention can simultaneously focus on actions with long durations and actions with short durations, thereby alleviating the problem of potential feature loss. Attached Figure Description

[0018] Figure 1 This is a flowchart of the present invention; Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings.

[0020] Reference Figure 1 The implementation steps of the embodiments of the present invention will be described in further detail below.

[0021] Step 1: Divide the original video stream into blocks.

[0022] The input video data is divided into consecutive frames using a sliding window method. Each window extracts a set of consecutive frames, and the window slides across the entire video at a preset time span to obtain a set of multiple consecutive frames of the current input video. Where, x k This represents the k-th consecutive frame group, where K represents the total number of consecutive frame blocks in the current input video.

[0023] The embodiments of this invention use the Thumos14 video dataset. For each input video in the Thumos14 dataset, image frames are first extracted at 30 FPS. A sliding window is used for partitioning, with a sliding window size of 16 and a window movement span of 4, to obtain a set of multiple consecutive frames of the current input video.

[0024] Step 2: Convert the image stream into a feature stream.

[0025] Each set of consecutive frames is sequentially input into the feature extraction model I3D to extract optical flow features and image features from the image video stream. The extracted two types of features are then horizontally concatenated to obtain the feature stream of that set. Among them, f k ∈R 256*2048 Let represent the feature stream of the k-th group, with a feature dimension of 2048 and a time dimension of 256. The image features of the k-th consecutive frame group have a feature dimension of 1024 and a time dimension of 256. The optical flow features of the k-th consecutive frame are represented by a feature dimension of 1024 and a time dimension of 256.

[0026] Step 3: Construct a multi-time-domain feature pyramid to fuse the extracted local information with global information.

[0027] The multi-temporal feature pyramid is composed of a multi-head attention module and a large-kernel convolution module connected in parallel. The large-kernel convolution module captures local feature dependency information in the feature stream through the convolutional receptive field, while the multi-head attention module captures global feature dependency information in the feature stream. The local feature dependency information and the global feature dependency information are added together and used as the output of the multi-temporal feature pyramid.

[0028] The large-kernel convolutional module comprises two feature extraction branches: an upper branch and a lower branch. The upper branch is a feature enhancement module, consisting of an average pooling layer with a kernel size of 3 and a stride of 1, a fully connected layer composed of 256 one-dimensional convolutions with a kernel size of 1, and a concatenation of the ReLU activation function. The lower branch is a feature extraction module, consisting of a parallel concatenation of a one-dimensional convolution with a kernel size of 1 and a large-kernel convolution with a kernel size of 9, followed by multiplication with a one-dimensional convolution with a kernel size of 1. The two branches sum the features to obtain the output dimension of the large-kernel convolutional module, which is 256*2048. To construct the pyramid structure, downsampling MaxPool is also used to obtain five scales with different time spans T = {256, 128, 64, 32, 16}.

[0029] The next branch is a multi-head attention module, where the input features and learnable parameters {W} are used. Q W K W V Perform dot products on each of the following, where the learnable parameter dimension is 256*256, and the product of {W} is... Q W K The calculated result is passed through a softmax layer to obtain a feature similarity matrix with dimensions 256*256, which is similar to {W}. VThe result of the multiplication is then multiplied by a dot product. This dot product is passed through a series of normalization layers (LN, dropout = 0.01), a multi-head attention layer (MSA), another LN, and a multilayer perceptron (MLP) to obtain the global feature dependency information of the input feature stream. k With dimensions of 256*2048, in order to construct the pyramid structure, MaxPool was used to obtain 5 scales with different time spans T={256,128,64,32,16}.

[0030] global feature dependency information Information dependent on local features The features are fused by feature addition and then used as the output of the multi-time-domain feature pyramid.

[0031] F a =Global + Local

[0032] Among them, F i a This is the output of the multi-time-domain feature pyramid.

[0033] The specific implementation steps are as follows:

[0034] Step 1, Feature Flow Local feature dependency information (Local) is obtained through the large kernel convolution module. k :

[0035]

[0036]

[0037] Where FC represents the fully connected layer mapping operation, implemented by one-dimensional convolution, Conv k This indicates the operation of extracting features using a large kernel convolution with a kernel size of 9. ReLU represents a non-linear mapping of the input. AvgPool represents the average pooling operation with a pooling window size of 3*3 and a stride of 1.

[0038] The second step is feature flow. After passing through a multi-head attention module, the input features and learnable parameters {W} are... Q W K W V Perform dot product on each of the following, where the product with {W} is... Q W K The calculated result is passed through a softmax layer to obtain the feature similarity, which is then compared with {W}. V The result of the multiplication is then multiplied by a dot product. This dot product is then passed through a cascaded normalization layer (LN), a multi-head attention layer (MSA), another normalization layer (LN), and a multilayer perceptron (MLP) to obtain the global feature dependency information of the input feature stream.k as follows:

[0039]

[0040]

[0041]

[0042] Where MLP stands for Multilayer Perceptron, which consists of multiple nonlinear mapping layers; LN represents layer normalization operation; MSA represents the use of a multi-head attention mechanism to extract comprehensive features; and W... Q W K W V This represents the learnable parameters Query, Key, and Value. The scaling factor is indicated by the superscript T, which signifies the transpose operation.

[0043] The third step is to integrate global feature dependency information. Information dependent on local features The features are fused together to form the output F of a multi-time-domain feature pyramid. i a :

[0044] F a =Global + Local

[0045] Where Global represents the global feature dependency information of the video extracted by the multi-head attention module, and Local represents the local feature dependency information of the video extracted by the large kernel convolution module.

[0046] Step 4: Construct the prominent frame feature pyramid.

[0047] input feature stream The original features are saliency mapping is performed using a one-dimensional convolutional layer with a kernel size of 1, a span of 1, and 256 kernels. Features are downsampled along the temporal dimension using MaxPool to obtain five different time spans T = {256, 128, 64, 32, 16}. This yields the saliency frame features of the input feature stream at each time span. The results are then horizontally stitched together to obtain...

[0048] The prominent frame feature pyramid will highlight the input feature stream The original features are saliency mapping is performed using a one-dimensional convolutional layer with a kernel size of 1 and a span of 1. Features are downsampled along the temporal dimension to obtain saliency frame features of the input feature stream at each time span. The results are then horizontally stitched together to obtain...

[0049]

[0050] Where layer represents the pyramid level, layer=1 indicates that this is the first pyramid level, and the original features need to undergo highlighting frame feature mapping operation. X i Let represent the input of the i-th layer of the pyramid, and MaxPool represent the max pooling operation.

[0051] Step 5: Construct a redundancy-eliminating pyramid model consisting of a multi-time-domain feature pyramid and a prominent frame feature pyramid connected in parallel.

[0052] The feature deduplication pyramid model is to highlight the feature output F of the feature pyramid of the frame. b The feature output F of the multi-time domain feature pyramid a After feature enhancement of the prominent frame through feature addition, the output F of the feature redundancy reduction pyramid model is obtained:

[0053] F = α * F a +β*F b ,

[0054] Where α represents the fusion factor of the multi-temporal-domain perception module pyramid, and β represents the fusion factor of the highlight frame retention strategy pyramid; the fusion factor is introduced to give the highlight frame a larger feature weight, thereby improving the feature highlighting effect.

[0055] Step 6: Train the feature redundancy reduction pyramid model.

[0056] The multi-layer features obtained from the prominent frame feature pyramid model are each placed into two branch structures. The first branch predicts the category of the action in the current frame, compares it with the actual action label in the video segment, and uses FocalLoss to calculate the classification loss. The second branch predicts the possible start and end times of the action and uses smoothed L1 loss to calculate the localization loss. The results of the two loss functions are summed and divided by the number of positive samples to obtain the training loss. The gradient backpropagation algorithm is used to iteratively update the model parameters until the network model loss function converges, resulting in a trained feature redundancy-reduction pyramid model.

[0057] The network model loss function is as follows: The generated location information is compared with the actual location labels in the video segment, and the location loss is calculated using smoothed L1 loss. The sum of the results calculated by the two loss functions is divided by the number of positive samples to obtain the training loss.

[0058]

[0059]

[0060]

[0061] Among them, L cls The difference between the model's predicted class labels and the true labels is represented by focus loss, L. reg The value λ represents the difference between the model's predicted time boundary and the actual time boundary, achieved through L1 smoothing loss. reg This represents the weight hyperparameter, which controls the degree to which the model focuses on the regression loss.

[0062] Step 7: Generate the location results.

[0063] Using the same method as steps 1 and 2, the feature stream of the video stream to be located is obtained. The feature stream is then input into a trained redundancy-reducing pyramid model, and the output is processed in blocks. After further transformation, the resulting feature stream is obtained. F i Let F represent the i-th video block. A complete video is divided into multiple video blocks, and these video blocks are horizontally spliced ​​together to form F. This feature F is input to the classification head and the localization head to obtain the action category and grid offset. The classification head is implemented by a one-dimensional convolutional layer with a kernel size of 3 and a number of kernels of 256, and an average pooling layer with a kernel size of 3 and a span of 3 connected in series.

[0064] The localization head is implemented using a one-dimensional convolutional layer with a kernel size of 1 and 256 kernels:

[0065] c t =ArgMax{Conv1D(F t )},

[0066] d st =Conv1D(F t ),

[0067] d et rConv1D(F t ),

[0068] Where, d st d represents the grid offset between the start time of the action at time t and time t. et This represents the grid offset between the action's end time at time t and t itself. The grid offset is in percentage format and needs to be mapped to the actual time. Mathematical calculations are used to map the predicted grid time to the actual time, and the resulting output, Y, is combined. t ={s t ,e t ,c t}:

[0069]

[0070]

[0071] Where t represents the current number of layers in the feature pyramid, with a total of t = {256, 128, 64, 32, 16} and 5 scales. k-1 This indicates the time step of the current layer (layer scaling ratio 2:1). This represents the start time of the action at time t within the Kth time span. This represents the end time of the action at time t within the Kth time span. Finally, Non-maximum Suppression (NMS) is used to obtain the optimal result.

[0072] The effects of this invention can be further illustrated by the following simulation experiments.

[0073] 1. Simulation conditions.

[0074] The simulation experiment of this invention uses PyCharm as the simulation tool. The computer configuration is as follows: motherboard: Gigabyte Z490 AORUS PRO AX, CPU: Intel(R) Core(TM) i7-10700KF CPU@3.80GHz, memory: 64GB, graphics card: Gigabyte GAMingOC GeForce RTX 3090*2, system: Ubuntu 24.04.

[0075] The simulation experiments used in this invention were conducted on the Thumos14 and MultiThumos14 video datasets. These datasets were created by a research team at Stanford University, primarily composed of researchers from academia and industry specializing in video understanding and computer vision. In the Thumos14 dataset, each segment is labeled with only a single action category. MultiThumos14 extends the original Thumos14 dataset by adding multi-action annotations, meaning that a given moment in the video can have multiple distinct action labels.

[0076] 2. Simulation experiment content and effect analysis.

[0077] The simulation experiments of this invention are three.

[0078] Simulation 1: The present invention and three existing technologies (ActionFormer algorithm, TemporalMaxer algorithm, and TriDet algorithm) were used to perform prediction simulations on Thumos14 data. The tIoU evaluation metric was used to evaluate the performance of the four methods at mAP@[0.3:0.1:0.7]. The performance evaluation results of the four methods are shown in Table 1.

[0079] The three existing technologies mentioned refer to:

[0080] The ActionFormer algorithm is an algorithm proposed by Stehouwer J et al. in their paper "Actionformer: Localizing moments of actions with transformers" (European Conference on Computer Vision (ECCV), pp 492-510, 2022), and is abbreviated as Actionformer algorithm.

[0081] The TemporalMaxer algorithm is an algorithm proposed by Tang TN et al. in their paper "Temporalmaxer: Maximize temporal context with only maxpooling for temporal action localization" (arXivpreprint arXiv, pp 202, 2022), and is abbreviated as TemporalMaxer algorithm.

[0082] The TriDet algorithm is an algorithm proposed by Shi D et al. in their paper "Tridet: Temporal action detection with relative boundary modeling" (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.pp18857-18866, 2023), and is abbreviated as TriDet algorithm.

[0083] Table 1. Summary of Thumos 14 Data Simulation Results

[0084] mAP This invention ActionFormer TemporalMaxer TriDet 0.3 84.3 82.1 82.8 83.6 0.4 80.5 77.8 78.9 80.1 0.5 73.8 71.0 71.8 72.9 0.6 62.2 59.4 60.5 62.4 0.7 48.0 43.9 44.7 47.4 Avg 69.7 66.8 67.7 69.3

[0085] Simulation 2: The present invention and three existing technologies (ActionFormer algorithm, TemporalMaxer algorithm, and TriDet algorithm) were used to perform prediction simulations on MultiTHUMOS14 data. The tIoU evaluation index was used to evaluate the performance at mAP@[0.2:0.2:0.7]. The results are shown in Table 2.

[0086] Table 2 Evaluation Table of MultiTHUMOS14 Data Simulation Results

[0087] mAP This invention ActionFormer TemporalMaxer TriDet 0.2 49.6 46.4 47.5 49.1 0.5 35.0 32.4 33.4 34.3 0.7 19.8 15.0 17.4 17.8 Avg 31.2 28.6 29.9 30.7

[0088] As shown in Table 2, the present invention outperforms the prior art in all AvgmAP evaluation metrics used in the experiment, demonstrating that the present invention can improve the results of temporal action localization compared with the prior art. Furthermore, MultiTHUMOS14 is an expanded version of the Thumos14 dataset, which has more detailed classifications and more action localization scenarios, demonstrating that the present invention can outperform existing methods in more difficult tasks.

[0089] The above results demonstrate that this invention can better locate the start and end times of actions in unedited videos. A feature pyramid architecture that preserves salient frames is used to reduce the impact of feature similarity caused by inter-layer operations on localization. Through a multi-temporal-domain awareness module, we fuse global and local temporal features for more comprehensive feature extraction. Experiments demonstrate that this invention achieves significant results on most datasets.

Claims

1. A feature deduplication temporal action localization method based on a prominent frame preservation strategy, characterized in that, A multi-temporal feature pyramid is constructed for extracting fusion of local and global information. A prominent frame preservation strategy is employed to extract prominent frame features from the original video using the prominent frame feature pyramid. The steps of this localization method include the following: Step 1: Divide the input raw video stream into blocks, and then convert the image video stream into a feature stream; Step 2: Construct a multi-time-domain feature pyramid, extract local information, and fuse it with global information. The multi-temporal feature pyramid is composed of a multi-head attention module and a large-kernel convolution module connected in parallel. The large-kernel convolution module captures local feature dependency information in the feature stream through the convolutional receptive field, while the multi-head attention module captures global feature dependency information in the feature stream. The local and global feature dependency information are added together to obtain the output of the multi-temporal feature pyramid. The large-kernel convolution module contains two feature extraction branches: the upper branch is a feature enhancement module, consisting of an average pooling layer, a fully connected layer, and an activation function concatenated; the lower branch is a feature extraction module, consisting of a one-dimensional convolution and a large-kernel convolution in parallel, which is then multiplied by a one-dimensional convolution. The two branches are combined to obtain the output of the large-kernel convolution module. The specific implementation steps are as follows: Step 1, Feature Flow Local feature dependency information (Local) is obtained through the large kernel convolution module. k : Where FC represents the fully connected layer mapping operation, implemented by one-dimensional convolution, Conv k This indicates the operation of extracting features using large kernel convolution with a kernel size of 9. ReLU indicates a non-linear mapping of the input. AvgPool indicates an average pooling operation with a pooling window size of 3*3 and a stride of 1. The second step is feature flow. After passing through a multi-head attention module, the input features and learnable parameters {W} are... Q W K W V Perform dot product on each of the following, where the product with {W} is... Q W K The calculated result is passed through a softmax layer to obtain the feature similarity, which is then compared with {W}. V The result of the multiplication is then multiplied by a dot product. This dot product is then passed through a cascaded normalization layer (LN), a multi-head attention layer (MSA), another normalization layer (LN), and a multilayer perceptron (MLP) to obtain the global feature dependency information of the input feature stream. k : Where MLP stands for Multilayer Perceptron, which consists of multiple nonlinear mapping layers; LN represents layer normalization operation; MSA represents the use of a multi-head attention mechanism to extract comprehensive features; and W... Q W K W V This represents the learnable parameters Query, Key, and Value. The superscript T represents the scaling factor, indicating the transpose operation; The third step is to integrate global feature dependency information. Information dependent on local features The features are fused together to form the output F of a multi-time-domain feature pyramid. i a : F a =Global+Local, Where Global represents the global feature dependency information of the video extracted by the multi-head attention module, and Local represents the local feature dependency information of the video extracted by the large kernel convolution module; Step 3: Emphasis frame preservation strategy is adopted. After performing one-dimensional convolution on the original feature stream to map the emphasis features, downsampling operation is performed only on the time span features. Emphasis frame features in the original video are extracted through the emphasis frame feature pyramid. Step 4: Construct a redundancy-eliminating pyramid model consisting of a multi-time-domain feature pyramid and a prominent frame feature pyramid connected in parallel; Step 5: Train the feature redundancy-reduction pyramid model; Step 6: Using the same method as in Step 1, obtain the feature stream of the video stream to be located, input the feature stream into the trained redundancy-reducing pyramid model, and output the localization result.

2. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, Step 1, which involves segmenting the original video stream into blocks, refers to dividing the input video data into consecutive frames using a sliding window method. Each window extracts a set of consecutive frames, and the window slides across the entire video at a preset time span to obtain a set of multiple consecutive frames of the current input video. Where, x k This represents the k-th consecutive frame group, where K represents the total number of consecutive frame blocks in the current input video.

3. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, The conversion of the image / video stream into a feature stream in step 1 refers to sequentially inputting each group of consecutive frames into the I3D feature extraction model to extract the optical flow features and image features of the image / video stream, and then horizontally concatenating the two types of features to obtain the feature stream of that group. Among them, f k Describes the characteristic flow of the k-th group. This represents the image features of the k-th consecutive frame. This represents the optical flow characteristics of the k-th consecutive frame.

4. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, The highlighted frame feature pyramid described in step 3 will input the feature stream. The original features are saliency mapping is performed using a one-dimensional convolutional layer with a kernel size of 1 and a span of 1. Features are downsampled along the temporal dimension to obtain saliency frame features of the input feature stream at each time span. The results are then horizontally stitched together to obtain... Where layer represents the pyramid level, layer=1 indicates that this is the first pyramid level, and the original features need to undergo highlighting frame feature mapping operation. X i Let represent the input of the i-th layer of the pyramid, and MaxPool represent the max pooling operation.

5. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, The feature deduplication pyramid model described in step 4 is to extract the feature output F of the salient frame feature pyramid. b The feature output F of the multi-time domain feature pyramid a After feature enhancement of the prominent frame through feature addition, the output F of the feature redundancy reduction pyramid model is obtained: F=α*F a +β*F b , Where α represents the fusion factor of the multi-temporal-domain sensing module pyramid, and β represents the fusion factor of the highlight frame retention strategy pyramid.

6. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, The steps for training the redundancy-reducing pyramid model described in step 5 are as follows: The multi-layer features obtained from the salient frame feature pyramid model are each placed into two branch structures. The first branch predicts the category, comparing it with the actual action label in the video segment and calculating the classification loss using focus loss. The second branch predicts the possible start and end times of the action. The model parameters are iteratively updated using the gradient backpropagation algorithm until the loss function of the entire network model converges, resulting in the trained redundancy-reducing pyramid model.

7. The feature deduplication temporal action localization method based on the salient frame preservation strategy according to claim 6, characterized in that, The loss function of the network model is as follows: Among them, L cls The difference between the model's predicted class label and the true label is represented by focus loss, L. reg The value λ represents the difference between the model's predicted time boundary and the actual time boundary, achieved through L1 smoothing loss. reg This represents the weight hyperparameter.

8. The feature deduplication temporal action localization method based on the salient frame retention strategy according to claim 1, characterized in that, The steps for generating the localization result in step 6 are as follows: input the feature stream into the trained redundancy-reducing pyramid model, process the output into blocks, and then transform the feature stream to obtain the desired feature stream. F i Let F represent the i-th video block. A complete video is divided into multiple video blocks, which are horizontally concatenated to form F. This feature F is input to the classification head and the localization head to obtain the action category and grid offset. The classification head is implemented by concatenating a one-dimensional convolutional layer with a kernel size of 3 and an average pooling layer with a kernel size of 3. The localization head is implemented by a one-dimensional convolutional layer with a kernel size of 1. The specific implementation formula is as follows: c t =ArgMax{Conv1D(F t )}, d st =Conv1D(F t ), d et =Conv1D(F t ), Where, d st d represents the grid offset between the start time of the action at time t and time t. et This represents the grid offset between the action end time at time t and t itself. The grid offset is in percentage format and needs to be mapped to the actual time. Mathematical calculations are used to map the predicted grid time to the actual time, and the resulting output Y is combined. t ={s t ,e t ,c t }; t represents the current layer number of the feature pyramid, 2 k-1 This indicates the time step of the current layer, with a layer scaling ratio of 2:

1. This represents the start time of the action at time t within the Kth time span. This represents the end time of the action at time t within the Kth time span. Finally, Non-maximum Suppression (NMS) is used to obtain the optimal result.

Images