A method for temporal action localization based on diffusion model
Through the timing action positioning method based on diffusion model, using multi-scale timing characterization of perceived TRP encoder and diffusion detector models, the problems of inaccurate boundary prediction and difficulty in timing modeling in the prior art are solved, and more accurate and robust action positioning effect.
Patent Information
- Application Number
- CN202410229845.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-02-29
AI Technical Summary
The existing timing action positioning methods mainly adopt direct learning strategies, ignoring the gradual nature of action boundaries, resulting in inaccurate boundaries prediction and difficult timing modeling, making it difficult to capture long-term dependence information of actions and timing evolution information on the timeline.
The timing action positioning method based on the diffusion model is adopted to extract long-term dependent timing feature information through multi-scale timing characterization perceived TRP encoder, and the diffusion detector model is used to gradually refine the prediction interval to reduce the error between the prediction and the real interval, so as to achieve Accurate boundary positioning.
It improves the accuracy and robustness of action positioning, can capture timing characteristics and long-term dependence information more precisely, alleviates the problems of blurred boundary and difficulty in timing modeling, and achieves more accurate action boundary positioning.
Smart Images

Figure CN118279786B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and in particular to a temporal action positioning method based on a diffusion model. Background Art
[0002] TAL (Temporal Action Localization) is a key task in the field of video understanding, which aims to classify and locate individual actions in unedited videos. The successful application of this task covers many fields, including video analysis, intelligent monitoring, and human-computer interaction, and has promoted the widespread application of video understanding technology in practical applications.
[0003] At present, the task of temporal action localization is mainly divided into two-stage and single-stage methods: The two-stage method completes action detection through two key stages: candidate proposal generation and classification. In the first stage, the model generates candidate action proposals, that is, time segments that may contain actions. This process can be achieved by adopting sliding windows or other proposal generation mechanisms. In the second stage, for each generated proposal, the model extracts the features of the local time segment and inputs the features of the proposal area into the action localization network to accurately predict the start and end time of the action. Some mainstream two-stage methods include S-CNN (Sequential Convolutional Neural Network), SSN (Structured Segment Network), BSN (Boundary Sensitive Network), BMN (Boundary-Matching Network), TAP, Dcan, etc. These methods have achieved remarkable achievements in the field of temporal action localization. However, since the two-stage method usually covers a more complex candidate proposal network and temporal modeling network, it has high computational resource requirements, resulting in relatively large computational overhead during training and inference.
[0004] The single-stage method is an early action localization strategy. The main goal of this type of method is to predict the start and end time of the action directly from the entire video sequence without the need for an explicit action proposal generation stage. Compared with the two-stage method, the single-stage method integrates candidate generation and classification into a single deep learning model, and directly predicts the temporal position of the action by directly modeling the entire video sequence. Some mainstream two-stage methods include SSAD (Single Shot Temporal Action Detection), Sap, GTAN, A2Net, ActionFormer, etc. This method omits the additional candidate proposal generation step, simplifies the action localization process, and helps to achieve more efficient training and convenient deployment.
[0005] However, most of them, whether adopting two-stage or single-stage methods, use a coarse or direct proposal predictor. These predictors involve directly localizing action boundaries in one go. However, due to the gradual nature of boundaries between actions, directly localizing action boundaries often leads to inaccuracies, especially in multi-scale video sequences. This direct localization strategy is not sensitive enough to the gradual boundary transition characteristics and usually leads to inaccurate results. Therefore, achieving accurate boundary localization remains a challenging task.
[0006] Temporal Action Localization (TAL) is a challenging vision task that aims to classify and localize all actions in untrimmed videos. A long-standing problem with TAL is the inaccurate boundary prediction due to the fuzzy action boundaries. Although existing methods have made efforts to address this issue in various aspects, most of them still adopt fixed or direct proposal learning strategies, making it difficult to achieve accurate localization.
[0007] (1) Intrinsic properties of action boundaries - boundary fuzziness: In terms of action boundary localization, existing methods usually use direct regression methods to locate action boundaries and use boundary features for refinement. These direct strategies have a problem in practice: their boundary predictions are imprecise due to the inherent characteristics of actions in videos. That is, unlike object detection, the action boundary in temporal action localization often appears as a gradual transition rather than a clear boundary. This gradual boundary transition poses a challenge to precise boundary localization, and most direct localization strategies are not sensitive to it. Intuitively, a gradual and refined boundary localization method can effectively alleviate boundary fuzziness and promote more precise boundaries.
[0008] (2) Detection-based TAL methods: Detection-based temporal action localization methods have emerged in recent years with the development of neural networks. Early segment-level methods attempted to use the global features of the predicted segments to locate boundaries to alleviate the ambiguity of boundaries. However, these methods may ignore the detailed information of each moment. On the other hand, instant-level methods are built on direct regression methods for boundary localization. However, this direct localization strategy is not sensitive enough to the gradual boundary transition characteristics, which usually leads to inaccurate results.
[0009] (3) Generation-based TAL methods: In recent years, diffusion models have become a hot topic in the field of computer vision, demonstrating impressive generation capabilities. It has been applied to various perception tasks such as segmentation, classification, detection, etc. These applications verify the versatility of diffusion in the field of perception and show that there are more potential applications worth exploring. Recently, DiffusionTAD attempted to build a diffusion-based target proposal system for temporal action prediction tasks. It uses a diffusion model to directly predict target values to achieve the generation of action boundaries. However, this design of directly generating targets, similar to direct localization, mainly relies on the number of iterative steps to refine the action boundaries, which may not be detailed enough, especially for long videos, resulting in inaccurate results.
[0010] (4) Temporal Context Modeling: Recent advances in temporal action localization have highlighted the importance of temporal context modeling (TCM) in capturing dependencies between features in videos. Previous studies have attempted to address this problem by exploring different approaches, including relation-aware pyramid networks, multi-stage CNNs, temporal context aggregation networks, and graph structures. These efforts have alleviated the problems of TCM to a certain extent. Over time, Transformer has demonstrated its powerful ability to model long-term dependencies between input sequences in machine translation tasks. Therefore, it has been naturally introduced into the task of temporal action localization and has become the preferred TCM solution for many researchers. For example, some methods use Transformer encoders to extract video representations to model sequence information. However, in recent years, in order to solve the rank loss and local similarity problems caused by the self-attention (SA) mechanism in Transformer, some methods have adopted convolution or pooling to replace SA to better model temporal information. Although these methods have achieved certain success in some aspects, they still have some defects, especially in capturing temporal evolution or key information, which may affect the temporal action localization results to a certain extent. Summary of the invention
[0011] In order to solve the problem that most of the existing temporal action localization algorithms use direct learning strategies to locate features, but ignore the fact that the transitions between different action boundaries in the video are gradual. In this case, direct detection has been proven to be insufficient to achieve accurate positioning of the action. In addition, the difficulty in temporal modeling is another challenge faced by current temporal positioning methods, which is manifested in that the evolution of actions in time usually involves complex changes, including different action stages, changes in speed, uncertainty in duration, etc. This makes it relatively difficult to accurately model and capture the temporal characteristics of actions. At present, the existing methods have technical problems in temporal modeling that it is difficult to learn the long-term dependency information of temporal features and the temporal evolution information on the timeline. The embodiment of the present invention provides a temporal action localization method and device based on a diffusion model. The technical solution is as follows:
[0012] On the one hand, a method for locating a time sequence action based on a diffusion model is provided, the method being implemented by a time sequence action locating device, the method comprising:
[0013] S1. Obtain video data to be processed.
[0014] S2. Input the video data into the temporal feature extractor to obtain the features of the video data.
[0015] S3. Input the features into the constructed multi-scale temporal representation-aware TRP encoder to obtain the long-term dependent temporal feature information of the video features.
[0016] S4. Input the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action localization result of the video data.
[0017] Optionally, the multi-scale temporal representation-aware TRP encoder in S3 includes: a first normalization layer, a temporal representation-aware TRP layer, a second normalization layer, and a multi-layer perceptron MLP.
[0018] The long-term dependence of video features is on temporal feature information, including long-term semantic information and temporal evolution information.
[0019] Optionally, the timing representation perception TRP layer includes: a timing evolution branch layer.
[0020] The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average features and the maximum features based on the output features, merge the average features and the maximum features to obtain the aggregate features, and obtain the time series evolution information of the aggregate features based on the aggregate features and the squeeze-incentive mechanism.
[0021] Optionally, the temporal representation perception TRP layer also includes: a temporal semantic branching layer.
[0022] The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and uses deep convolution and 1×1 convolution to extract information from the output features to obtain the long-term semantic information of the video features.
[0023] Optionally, the training process of the diffusion detector model in S4 includes:
[0024] The long-term dependent temporal feature information and the true interval after noise addition are input into the Transformer decoder of the diffusion detector model to learn the error distribution value between the true interval after noise addition and the Gaussian noise.
[0025] The error between the true interval after noise addition and Gaussian noise is gradually reduced by using the mean square error, thereby forcing the true interval after noise addition to gradually approach pure Gaussian noise.
[0026] As the error loss function gradually decreases, a trained network model is obtained.
[0027] Optionally, in S4, the long-term dependent temporal feature information is input into a trained diffusion detector model to obtain a temporal action localization result of the video data, including:
[0028] S41. Input the long-term dependent temporal feature information, random Gaussian noise and the number of iteration steps into the trained diffusion detector, and generate the error value Δf between the prediction interval and the target interval in each step through the Transformer decoder of the trained diffusion detector.
[0029] S42, subtract the error value Δf from the predicted interval value to obtain the interval prediction value for the next step.
[0030] S43. The interval prediction value of the next step is continuously updated according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval is gradually reduced, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
[0031] On the other hand, a device for locating a time sequence action based on a diffusion model is provided, and the device is applied to a method for locating a time sequence action based on a diffusion model, and the device comprises:
[0032] The acquisition module is used to acquire the video data to be processed.
[0033] The feature extraction module is used to input the video data into the temporal feature extractor to obtain the features of the video data.
[0034] The input module is used to input the features into the constructed multi-scale temporal representation-aware TRP encoder to obtain the long-term dependent temporal feature information of the video features.
[0035] The output module is used to input the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action localization result of the video data.
[0036] Optionally, a multi-scale temporal representation-aware TRP encoder includes: a first normalization layer, a temporal representation-aware TRP layer, a second normalization layer, and a multi-layer perceptron MLP.
[0037] The long-term dependence of video features is on temporal feature information, including long-term semantic information and temporal evolution information.
[0038] Optionally, the timing representation perception TRP layer includes: a timing evolution branch layer.
[0039] The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average features and the maximum features based on the output features, merge the average features and the maximum features to obtain the aggregate features, and obtain the time series evolution information of the aggregate features based on the aggregate features and the squeeze-incentive mechanism.
[0040] Optionally, the temporal representation perception TRP layer also includes: a temporal semantic branching layer.
[0041] The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and uses deep convolution and 1×1 convolution to extract information from the output features to obtain the long-term semantic information of the video features.
[0042] Optionally, the output module is further configured to:
[0043] The long-term dependent temporal feature information and the true interval after noise addition are input into the Transformer decoder of the diffusion detector model to learn the error distribution value between the true interval after noise addition and the Gaussian noise.
[0044] The error between the true interval after noise addition and the Gaussian noise is gradually reduced by using the mean square error, thereby forcing the true interval after noise addition to gradually approach the pure Gaussian noise.
[0045] As the error loss function gradually decreases, a trained network model is obtained.
[0046] Optionally, the output module is further configured to:
[0047] S41. Input the long-term dependent temporal feature information, random Gaussian noise and the number of iteration steps into the trained diffusion detector, and generate the error value Δf between the prediction interval and the target interval in each step through the Transformer decoder of the trained diffusion detector.
[0048] S42, subtract the error value Δf from the predicted interval value to obtain the interval prediction value for the next step.
[0049] S43. The interval prediction value of the next step is continuously updated according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval is gradually reduced, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
[0050] On the other hand, a time sequence action positioning device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the time sequence action positioning methods based on the diffusion model is implemented.
[0051] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned sequential action positioning methods based on the diffusion model.
[0052] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0053] In the embodiment of the present invention, the task of locating sequential actions can be solved by a generative method.
[0054] A progressive and refinement method is designed. The progressive nature of the diffusion model, coupled with the gradual refinement of the introduced error value Δf, makes accurate boundary localization feasible.
[0055] A temporal modeling method, the TRP module, is introduced to capture the temporal evolution information and long-term dependency information of features by strengthening temporal modeling, thereby further improving the accuracy of action localization. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0057] Figure 1 is a flow chart of a method for positioning a time sequence action based on a diffusion model provided by an embodiment of the present invention;
[0058] Figure 2 is a schematic diagram of the structure of a multi-scale TRP encoder provided by an embodiment of the present invention;
[0059] Figure 3 is a TRP module architecture diagram provided by an embodiment of the present invention;
[0060] Figure 4 is a schematic diagram of a forward diffusion structure provided by an embodiment of the present invention;
[0061] Figure 5 is a schematic diagram of a reverse denoising structure provided by an embodiment of the present invention;
[0062] Figure 6 is a comparison result diagram on the THUMOS14 dataset provided by an embodiment of the present invention;
[0063] Figure 7 is a comparison result diagram on the EPIC-KITCHEN 100 data set provided by an embodiment of the present invention;
[0064] Figure 8 It is a comparison result diagram on the Actiynet-1.3 data set provided by an embodiment of the present invention;
[0065] Fig. 9 It is a general model architecture diagram provided by an embodiment of the present invention;
[0066] Fig.10 It is a block diagram of a sequential action positioning device based on a diffusion model provided by an embodiment of the present invention;
[0067] Fig.11 It is a structural schematic diagram of a sequential motion positioning device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0068] The technical solution of the present invention is described below in conjunction with the accompanying drawings.
[0069] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.
[0070] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.
[0071] In the embodiments of the present invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.
[0072] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0073] The embodiment of the present invention provides a method for locating a time sequence action based on a diffusion model. The method can be implemented by a time sequence action locating device, which can be a terminal or a server. Figure 1 The flowchart of the method for localizing temporal actions based on a diffusion model is shown. The processing flow of the method may include the following steps:
[0074] S1. Obtain video data to be processed.
[0075] S2. Input the video data into the temporal feature extractor to obtain the features of the video data.
[0076] In a feasible implementation manner, using a temporal feature extractor to extract features from video data is a prior art and is not limited here.
[0077] S3. Input the features into the constructed multi-scale temporal representation-aware TRP encoder to obtain the long-term dependent temporal feature information of the video features.
[0078] In a feasible implementation, Figure 2 As shown, the present invention captures the long-term dependency and temporal evolution information of multi-scale features through a multi-scale TRP module.
[0079] Optionally, the multi-scale temporal representation-aware TRP encoder in S3 includes: a first normalization layer, a temporal representation-aware TRP layer, a second normalization layer, and a multi-layer perceptron MLP. Regarding the TRP layer, it specifically includes a temporal evolution branch and a temporal semantic branch. The temporal evolution branch captures the dynamic evolution of the temporal sequence and enhances the discriminability of the features by merging double pooling operations and utilizing squeezing and extraction designs. At the same time, the temporal semantic branch captures long-term semantic information by expanding the receptive field of the convolution kernel. Combining these two branches improves the temporal modeling of the feature model.
[0080] The long-term dependence of video features is on temporal feature information, including long-term semantic information and temporal evolution information.
[0081] In one feasible implementation, the multi-scale temporal representation-aware TRP encoder is a simple and effective way to temporally model video features to capture long-term semantic and evolutionary information, thereby improving the detection performance of the model. First, it encodes the video features and downsamples the feature pyramid multiple times through max pooling to form a multi-scale feature representation, like most temporal action localization algorithms. In this process, the features of each pyramid layer are processed using a Transformer-based module. In this module, its self-attention (SA) layer is replaced by the proposed TRP (Temporal Relationship Preservation) layer, and the rest of the structure remains unchanged.
[0082] Furthermore, the design of the TRP layer is crucial for temporal modeling, and it is divided into two branches: the Temporal Evolution Branch (TEB) and the Temporal Semantic Branch (TSB) modules, as shown in Figure 3 shown.
[0083] Optionally, the timing representation perception TRP layer includes: a timing evolution branch layer.
[0084] The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average features and the maximum features based on the output features, merge the average features and the maximum features to obtain the aggregate features, and obtain the time series evolution information of the aggregate features based on the aggregate features and the squeeze-incentive mechanism.
[0085] In a feasible implementation, the temporal evolution branch: From the perspective of the channel, our focus is to capture the temporal evolution and aggregate information while enhancing the discriminability of local features. To achieve this goal, we adopt a double pooling method to further reduce the local similarity by merging the average and maximum features at the video level. The average pooling method is usually used to integrate information to make the overall feature map smoother, thereby obtaining global information. However, we found that selecting representative information through maximum pooling is more advantageous in reducing local similarity and enhancing the distinguishability of features. Therefore, we choose to merge the average and maximum dual pooling features at the same time to aggregate feature information. At the same time, the present invention introduces a squeeze-excitation mechanism to learn the temporal evolution of features along the time axis, so that the network is more sensitive to the dynamic changes of video sequences. That is, the temporal features are compressed by average pooling and maximum pooling to obtain global features and local features. Subsequently, these features are activated through weight distribution to increase the responsiveness of the network to the dynamic changes of video sequences.
[0086] Optionally, the temporal representation perception TRP layer also includes: a temporal semantic branching layer.
[0087] The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and uses deep convolution and 1×1 convolution to extract information from the output features to obtain the long-term semantic information of the video features.
[0088] In a feasible implementation, the temporal semantic branch: In addition to capturing the dynamic evolution of temporal features, ensuring the capture of semantic information remains an important consideration in temporal modeling. The present invention expands the receptive field of the network by expanding the convolution kernel, thereby capturing the long-term semantic information of temporal features. Inspired by recent algorithms and large-kernel convolutions, the present invention uses deep convolution to model large-kernel convolutions. By introducing deep convolutions, the present invention is able to capture semantic content from a wider receptive field and reduce the number of channels by using 1×1 convolutions, thereby reducing the computational burden. Overall, the combined strategy of deep convolution and 1×1 convolution effectively captures long-term semantic dependencies.
[0089] In short, the temporal evolution branch captures temporal dynamics and enhances the discriminability of features by incorporating double pooling and squeeze-excite design. Meanwhile, the temporal semantic branch effectively captures long-term semantic information by expanding the receptive field of the convolution kernel.
[0090] The present invention uses a multi-scale temporal representation perceptual encoder to construct pyramid features, and then introduces the corresponding features into a diffusion detector with shared weights. At a small scale, the prediction results of each frame and each step are refined to gradually narrow the gap between the prediction and the true value.
[0091] S4. Input the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action localization result of the video data.
[0092] In a feasible implementation manner, the progressive diffusion detector of the present invention gradually refines the prediction interval frame by frame in a progressive manner, so that the prediction interval gradually approaches the true interval.
[0093] Optionally, the training process of the diffusion detector model in S4 includes:
[0094] The long-term dependent temporal feature information and the true interval after noise addition are input into the Transformer decoder of the diffusion detector model to learn the error distribution value between the true interval after noise addition and the Gaussian noise.
[0095] The error between the true interval after noise addition and the Gaussian noise is gradually reduced by using the mean square error, thereby forcing the true interval after noise addition to gradually approach the pure Gaussian noise.
[0096] As the error loss function gradually decreases, a trained network model is obtained.
[0097] In a feasible implementation, Figure 4 As shown, in the forward diffusion process, the present invention gradually adds noise to the target interval until it eventually becomes complete noise. For the target interval, a total of T steps of diffusion process are included. The distribution of the interval to the noise interval of each step is obtained by adding Gaussian noise to the data of the previous step, which provides support for the subsequent reverse process. For the training phase, the diffusion model with learnable parameters learns a mapping: predicted noise to pure noise. The mapping is a neural network. The network is trained by using a mean square error loss function, that is, minimizing the difference between the predicted noise and the true noise. As the error loss function gradually decreases, a trained network is obtained.
[0098] Optionally, the step of inputting the long-term dependent temporal feature information into the trained diffusion detector model in S4 to obtain the temporal action localization result of the video data may include S41-S43:
[0099] S41. Input the long-term dependent temporal feature information, random Gaussian noise and the number of iteration steps into the trained diffusion detector, and generate the error value Δf between the prediction interval and the target interval in each step through the Transformer decoder of the diffusion detector.
[0100] S42, subtract the error value Δf from the predicted interval value to obtain the interval prediction value for the next step.
[0101] S43. The interval prediction value of the next step is continuously updated according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval is gradually reduced, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
[0102] In a feasible implementation, Figure 5 As shown in Figure 1, the forward process of diffusion converts the target interval into a noise interval, while the reverse process is the denoising process. If the distribution of each step is known, starting from random noise, sample data close to the true value can be generated by stepwise denoising. The present invention uses a trained neural network to estimate the distribution. First, the feature F output by the multi-scale TRP encoder, the random noise x t The number of iterations t is sent to the diffusion detector. Then, the error value Δf between the predicted value and the true value is generated through the Transformer decoder. Next, the prediction result of this step is subtracted from the generated error value Δf to obtain the prediction result of the next step. Finally, the DDIM denoising diffusion implicit model (DenoisingDiffusion Implicit Models) algorithm is used to gradually refine the prediction value by continuously updating the generated Δf. After T iterations, a prediction value close to the true interval can be obtained.
[0103] This error value Δf captures the distribution difference between the predicted value and the target value at each step. By continuously reducing this difference, the predicted value will gradually fit the target value. This design of gradually refining the prediction is consistent with the gradual nature of action boundaries and provides a robust method for accurate boundary localization in videos.
[0104] Further, model training: The training phase involves converting the target interval into the noise interval, using the network to learn the distribution gap between the target interval and the noise. Subsequently, the loss function is used to gradually narrow the gap and finally generate the noise interval.
[0105] Model inference: The model starts with noise sampled from a Gaussian distribution and gradually refines the prediction. The purpose of this inference process is to iteratively update a small error value, gradually refine the prediction at each step, and ensure that the predicted result gradually approaches the true value.
[0106] Figure 6 The comparison results between the performance of the present invention and the state-of-the-art methods on the THUMOS14 dataset are provided. Compared with other methods that directly generate target intervals, the progressive and refined design of the present invention enables the model to achieve an average mean average precision (mAP) of 69.3%, which exceeds the compared methods. The experimental results verify the effectiveness of the progressive diffusion design of the present invention and demonstrate the potential to alleviate the problems associated with fuzzy boundaries.
[0107] like Figure 7 As shown in Figure 2, for the EPIC-KITCHEN 100 dataset, our method outperforms the compared methods with mAPs of 25.4% and 23.2% for verbs and nouns, respectively. These results highlight the excellent performance of our method on this challenging dataset.
[0108] like Figure 8 As shown in the figure, on the ActivityNet-1.3 dataset, the algorithm of the present invention achieves 36.9% in mAP, second only to TCANet, which may be due to the use of more powerful features by TCANet. In general, the method of the present invention shows comparable results with existing methods on this challenging dataset, showing its effectiveness on the ActivityNet-1.3 dataset.
[0109] Summary: Extensive experiments on the THUMOS14, EPIC-KITCHEN 100, and Actiynet-1.3 datasets show that the proposed method achieves comparable performance with advanced methods. The proposed algorithm provides a feasible solution to the boundary fuzziness problem and action boundary generation, and provides new directions and ideas for further research on temporal action localization.
[0110] The overall architecture of the model is as follows Fig. 9 As shown, it includes feature input, multi-scale TRP encoder ( Fig. 9 (a)), forward diffusion ( Fig. 9 (b)) and reverse denoising ( Fig. 9 (c)) and other parts.
[0111] Furthermore, the workflow of the model can be described as follows: First, the features of the input video are extracted using a temporal feature extractor. Next, the present invention constructs a multi-scale TRP encoder to capture the long-term dependent temporal feature information in the video. Then, the diffusion detector model is trained through forward diffusion. Finally, starting from full noise, the prediction results are continuously refined through the trained model until the target is finally detected.
[0112] In order to alleviate the problems existing in the prior art, the present invention proposes a progressive and refined temporal action localization method based on a diffusion model. The method achieves accurate boundary localization by implementing a progressive diffusion detector, which is designed to iteratively refine the prediction by updating the error offset (Δf). The adjustment of Δf gradually reduces the distance between the predicted value and the target value, forcing the predicted value to become more accurate when it is close to the target value. This coarse-to-fine paradigm captures the distribution difference between the predicted value and the target value. It enhances the robustness of the output and ensures the accuracy and stability of video localization under different inputs. Accurate action localization also relies on appropriate temporal context modeling TCM to support effective temporal information for localization. Therefore, the present invention introduces a new TCM module, the Temporal Representation Perception (TRP) layer to enhance feature perception. This module skillfully captures temporal evolution and long-term context information by introducing squeeze and excitation design and large convolution kernels.
[0113] analyze:
[0114] (1) Compared with other detection-based TAL methods: The present invention fully considers the intrinsic properties of the progressive nature of action boundaries and proposes a progressive and refined positioning strategy. This strategy makes full use of the progressive characteristics of the diffusion model and improves the positioning accuracy by refining the prediction process. Specifically, the present invention uses the error value Δf between the predicted value and the target value generated by the prediction to gradually narrow the gap between the predicted value and the target value using this tiny error value. This design enables the network to capture the tiny distribution differences between the predicted value and the target value, and by continuously narrowing these differences, it ultimately achieves accurate positioning of the action boundary. This not only improves the robustness of the output results, but also enhances the stability when processing videos of different scales.
[0115] (2) Comparison with generation-based DiffusionTAD: DiffusionTAD attempts to establish a diffusion-based target proposal system for temporal action prediction tasks, by using a diffusion model to directly predict target values to achieve action boundary generation. However, this design of directly generating targets, similar to direct localization, mainly relies on iterative steps to refine action boundaries, which may not be detailed enough, especially for long video sequences, and may lead to inaccurate results. In contrast, the method of the present invention avoids directly predicting target proposals in each step. The present invention adopts an improved gradually refined design to achieve accurate action localization by gradually narrowing the gap between the predicted results and the target values of each step on a smaller scale. This coarse-to-fine design process plays a vital role in the overall method of the present invention and improves the accuracy of action localization.
[0116] In the embodiment of the present invention, the task of locating sequential actions can be solved by a generative method.
[0117] A progressive and refinement method is designed. The progressive nature of the diffusion model, coupled with the gradual refinement of the introduced error value Δf, makes accurate boundary localization feasible.
[0118] A temporal modeling method, the TRP module, is introduced to capture the temporal evolution information and long-term dependency information of features by strengthening temporal modeling, thereby further improving the accuracy of action localization.
[0119] Fig.10 1 is a block diagram of a device for positioning a time sequence action based on a diffusion model according to an exemplary embodiment. The device is used in a method for positioning a time sequence action based on a diffusion model. Fig.10 The device includes an acquisition module 1010, a feature extraction module 1020, an input module 1030, and an output module 1040.
[0120] in:
[0121] The acquisition module 1010 is used to acquire video data to be processed.
[0122] The feature extraction module 1020 is used to input the video data into the temporal feature extractor to obtain the features of the video data.
[0123] The input module 1030 is used to input the features into the constructed multi-scale temporal representation-aware TRP encoder to obtain the long-term dependent temporal feature information of the video features.
[0124] The output module 1040 is used to input the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action positioning result of the video data.
[0125] Optionally, a multi-scale temporal representation-aware TRP encoder includes: a first normalization layer, a temporal representation-aware TRP layer, a second normalization layer, and a multi-layer perceptron MLP.
[0126] The long-term dependence of video features is on temporal feature information, including long-term semantic information and temporal evolution information.
[0127] Optionally, the timing representation perception TRP layer includes: a timing evolution branch layer.
[0128] The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average features and the maximum features based on the output features, merge the average features and the maximum features to obtain the aggregate features, and obtain the time series evolution information of the aggregate features based on the aggregate features and the squeeze-incentive mechanism.
[0129] Optionally, the temporal representation perception TRP layer also includes: a temporal semantic branching layer.
[0130] The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and uses deep convolution and 1×1 convolution to extract information from the output features to obtain the long-term semantic information of the video features.
[0131] Optionally, the output module is further configured to:
[0132] The long-term dependent temporal feature information and the true interval after noise addition are input into the Transformer decoder of the diffusion detector model to learn the error distribution value between the true interval after noise addition and the Gaussian noise.
[0133] The error between the true interval after noise addition and Gaussian noise is gradually reduced by using the mean square error, thereby forcing the true interval after noise addition to gradually approach pure Gaussian noise.
[0134] As the error loss function gradually decreases, a trained network model is obtained.
[0135] Optionally, the output module is further configured to:
[0136] S41. Input the long-term dependent temporal feature information, random Gaussian noise and the number of iteration steps into the trained diffusion detector, and generate the error value Δf between the prediction interval and the target interval in each step through the Transformer decoder of the diffusion detector.
[0137] S42, subtract the error value Δf from the predicted interval value to obtain the interval prediction value for the next step.
[0138] S43. The interval prediction value of the next step is continuously updated according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval is gradually reduced, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
[0139] In the embodiment of the present invention, the task of locating sequential actions can be solved by a generative method.
[0140] A progressive and refined method is designed. The progressive design of the diffusion model, coupled with the gradual refinement of the introduced error value Δf, makes accurate boundary localization feasible.
[0141] A temporal modeling method, the TRP module, is introduced to capture the temporal evolution information and long-term dependency information of features by strengthening temporal modeling, thereby further improving the accuracy of action localization.
[0142] Fig.11 is a schematic diagram of the structure of a sequential motion positioning device provided by an embodiment of the present invention, such as Fig.11 As shown, the timing action positioning device may include the above Fig.10 The time sequence action positioning device based on the diffusion model is shown. Optionally, the time sequence action positioning device 1110 may include a first processor 2001.
[0143] Optionally, the sequential motion positioning device 1110 may further include a memory 2002 and a transceiver 2003 .
[0144] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.
[0145] Combine the following Fig.11 The components of the sequential motion positioning device 1110 are specifically introduced as follows:
[0146] The first processor 2001 is the control center of the sequential motion positioning device 1110, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).
[0147] Optionally, the first processor 2001 can execute various functions of the sequential motion positioning device 1110 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .
[0148] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig.11 CPU0 and CPU1 are shown in FIG.
[0149] In a specific implementation, as an embodiment, the time sequence motion positioning device 1110 may also include multiple processors, such as Fig.11 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0150] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.
[0151] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be located through the interface circuit ( Fig.11 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0152] The transceiver 2003 is used to communicate with a network device or a terminal device.
[0153] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig.11 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0154] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and locate the interface circuit ( Fig.11 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.
[0155] It should be noted that Fig.11 The structure of the timing action positioning device 1110 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0156] In addition, the technical effects of the time-series action positioning device 1110 can refer to the technical effects of the time-series action positioning method based on the diffusion model described in the above method embodiment, which will not be repeated here.
[0157] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0158] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0159] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0160] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.
[0161] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0162] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0163] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0165] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0167] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0168] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0169] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A method for temporal action localization based on diffusion model, It is characterized in that The method comprises: S1. Obtain video data to be processed; S2, inputting the video data into a temporal feature extractor to obtain features of the video data; S3, inputting the features into the constructed multi-scale temporal representation-aware TRP encoder to obtain long-term dependent temporal feature information of the video features; S4, inputting the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action positioning result of the video data; The multi-scale temporal representation perception TRP encoder in S3 includes: a first normalization layer, a temporal representation perception TRP layer, a second normalization layer and a multi-layer perceptron MLP; The long-term dependent temporal feature information of the video features, including long-term semantic information and temporal evolution information; The timing representation perception TRP layer includes: a timing evolution branch layer; The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average feature and the maximum feature according to the output features, merge the average feature and the maximum feature to obtain the aggregate feature, and obtain the time series evolution information of the aggregate feature according to the aggregate feature and the squeeze-incentive mechanism; The temporal representation perception TRP layer also includes: a temporal semantic branching layer; The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and use deep convolution and 1×1 convolution to extract information from the output features to obtain long-term semantic information of the video features.
2. The method for localizing temporal actions based on a diffusion model according to claim 1, It is characterized in that The training process of the diffusion detector model in S4 includes: The long-term dependent temporal feature information and the true interval after noise addition are input into the Transformer decoder of the diffusion detector model to learn the error distribution value between the true interval after noise addition and the Gaussian noise. By using the mean square error to gradually reduce the error between the true interval after adding noise and the Gaussian noise, the true interval after adding noise is forced to gradually approach the pure Gaussian noise. As the error loss function gradually decreases, a trained network model is obtained.
3. The method for localizing temporal actions based on a diffusion model according to claim 1, It is characterized in that The step of inputting the long-term dependent temporal feature information into the trained diffusion detector model to obtain the temporal action localization result of the video data in S4 includes: S41, inputting the long-term dependent time series feature information, random Gaussian noise and the number of iteration steps into a trained diffusion detector, and generating an error value Δf between a prediction interval and a target interval in each step through a Transformer decoder of the trained diffusion detector; S42, subtracting the error value Δf from the prediction interval value to obtain the interval prediction value for the next step; S43, continuously updating the interval prediction value of the next step according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval gradually decreases, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
4. A time sequence action positioning device based on a diffusion model, the time sequence action positioning device based on a diffusion model is used to implement the time sequence action positioning method based on a diffusion model as claimed in any one of claims 1 to 3, It is characterized in that The device comprises: An acquisition module, used for acquiring video data to be processed; A feature extraction module, used for inputting the video data into a temporal feature extractor to obtain features of the video data; An input module, used for inputting the features into a constructed multi-scale temporal representation perception TRP encoder to obtain long-term dependent temporal feature information of video features; An output module, used for inputting the long-term dependent temporal feature information into a trained diffusion detector model to obtain a temporal action localization result of the video data; The multi-scale temporal representation perception TRP encoder comprises: a first normalization layer, a temporal representation perception TRP layer, a second normalization layer and a multi-layer perceptron MLP; The long-term dependent temporal feature information of the video features, including long-term semantic information and temporal evolution information; The timing representation perception TRP layer includes: a timing evolution branch layer; The time series evolution branch layer is used to obtain the output features of the first normalization layer, obtain the average feature and the maximum feature according to the output features, merge the average feature and the maximum feature to obtain the aggregate feature, and obtain the time series evolution information of the aggregate feature according to the aggregate feature and the squeeze-incentive mechanism; The temporal representation perception TRP layer also includes: a temporal semantic branching layer; The temporal semantic branch layer is used to obtain the output features of the first normalization layer, and use deep convolution and 1×1 convolution to extract information from the output features to obtain long-term semantic information of the video features.
5. The time sequence action positioning device based on the diffusion model according to claim 4, It is characterized in that The output module is used for: S41, inputting the long-term dependent time series feature information, random Gaussian noise and the number of iteration steps into a trained diffusion detector, and generating an error value Δf between a prediction interval and a target interval in each step through a Transformer decoder of the trained diffusion detector; S42, subtracting the error value Δf from the prediction interval value to obtain the interval prediction value for the next step; S43, continuously updating the interval prediction value of the next step according to the denoising diffusion implicit model DDIM algorithm, so that the error value between the prediction interval and the target interval gradually decreases, thereby forcing the prediction interval value to gradually approach the target interval value, and obtaining the temporal action positioning result of the video data.
6. A time sequence motion positioning device, It is characterized in that The sequential action positioning device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.
7. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Sequence action positioning method, system and device based on set prediction and medium
CN114708523A
Video space-time action positioning method based on progressive attention hypergraph
CN114882403A
Video time sequence action positioning method based on semantic-level time sequence correlation modeling
CN115471771A
Time sequence action positioning method based on global perception and nomination relation mining
CN115497022A
Conditional diffusion model-based time series data prediction method and system
CN117076931A
