First view angle online action segmentation method based on uncertainty

Abstract

The invention discloses a first view angle online action segmentation method based on uncertainty, and relates to the technical field of video action analysis. According to the method, a strategy of dynamically changing the network according to uncertainty is adopted, and efficient and accurate online action segmentation under the condition of information loss in the future is realized. According to the invention, a method for scenario context according to an uncertainty principle is adopted. The method comprises two key parts: 1, a context aggregation converter which combines a joint auto-encoder and an auto-regression design to enhance contextualization of frame-level features and effectively overcome limitations caused by lack of subsequent contexts in a streaming video; and 2, action contextualization based on evidence: quantifying uncertainty in a model decision by using evidence deep learning, and correcting fuzzy prediction based on a previous context. According to the mode, the context scene information is supplemented, and the problems in the prior art can be effectively solved. The method can be widely applied to scenes related to first person view angle video action analysis, such as human body behavior monitoring based on wearable equipment, character action understanding in augmented reality and virtual reality scenes and the like.

Description

Technical Field

[0001] The present invention belongs to the field of first-perspective online action segmentation in action segmentation, and particularly relates to online recognition and segmentation of continuous actions in a video stream. Background Art

[0002] In today's rapidly developing digital age, robotics and smart wearable devices are booming, driving a surge in demand for first-person perspective unedited video understanding. As these technologies continue to evolve, they've penetrated numerous fields. From everyday smart home control, where people can control appliances via voice commands, which involves capturing and analyzing user movements and voice from a first-person perspective, to interactive remote learning in education, where teachers can better understand students' learning status and operational processes through first-person perspective videos.

[0003] The widespread application of this technology has not only changed the way we acquire and process information, but also made it extremely important to efficiently process and analyze unedited video data from a first-person perspective. This demand is particularly prominent in action-related application scenarios. First-person video data contains rich action details and environmental information, which is critical for interpreting user intentions, analyzing behavioral patterns, and achieving real-time interaction. However, compared with traditional offline action segmentation tasks, online action segmentation from a first-person perspective faces unique challenges. It requires the model to process every frame in the video stream under strict real-time conditions, proceeding strictly in chronological order, and unable to obtain information from subsequent frames. This means that the model must quickly and accurately identify and segment continuous action clips under limited conditions, which is crucial for achieving application goals such as smooth human-computer collaboration, intelligent security monitoring, and immersive virtual reality experiences.

[0004] In the specific scenario of online action segmentation, the model can only rely on the current observation and past contextual information to interpret the content of each frame in real time. This real-time nature requires the model to possess not only efficient computing power but also excellent feature extraction and decision-making capabilities. However, due to the unique nature of online action segmentation, traditional action segmentation methods face two major challenges. First, first-person videos are captured by robots or humans wearing wearable devices, rather than traditional surveillance equipment, resulting in a highly dynamic and limited field of view. For example, in real-world scenarios, human movement can rapidly change the perspective, as shown in the figure. This makes it difficult for the model to consistently and stably interpret the visual content of each frame, preventing it from obtaining a comprehensive view of the activity as with traditional surveillance video. Second, the real-time nature of online action segmentation requires the model to strictly rely on the current observation and previous contextual information to process each frame in real time. This real-time nature places high demands on the model, requiring not only efficient computing power to quickly process large amounts of data, but also strong feature extraction and decision-making capabilities to make accurate judgments in a short period of time.

[0005] Online action segmentation is an emerging field that has been inherited from the field of action segmentation. Researchers are actively exploring various methods, such as introducing attention mechanisms, using recurrent neural networks or long short-term memory networks to capture temporal dependencies, and using transfer learning and domain adaptation techniques to improve the generalization ability of the model. However, most of these methods are only applicable to offline action segmentation. In the field of first-person perspective online action segmentation, due to the highly dynamic and limited field of view and the lack of future information, previous methods have struggled to achieve good results. In view of this, we propose an uncertainty-based first-person perspective online action segmentation method, which adopts a strategy of dynamically adjusting the network based on uncertainty to achieve efficient and accurate online action segmentation in the absence of future information.

[0006] In summary, online action segmentation, as an emerging research field, integrates the latest advances in computer vision, machine learning, and artificial intelligence, aiming to enhance the understanding and analysis of first-person perspective video data. With continued technological innovation and increasing application demand, we expect to see more innovative achievements and breakthroughs in this field, providing users with more intelligent and personalized interactive experiences. In the future, we expect to see widespread application of online action segmentation technology in a wide range of fields, including intelligent assistants, health monitoring, and autonomous driving. Summary of the Invention

[0007] The purpose of this invention is to address the problems of existing first-person perspective online action segmentation methods, and to develop an online action segmentation network structure based on uncertainty, adopting a strategy of dynamically changing the network according to uncertainty, so as to achieve efficient and accurate online action segmentation in the absence of future information.

[0008] I3D Feature Extractor: I3D (Inflated 3D ConvNet) is a deep learning model for video analysis. It simultaneously captures spatial and temporal features of videos by extending the convolution and pooling kernels of a 2D convolutional neural network (CNN) to 3D. This approach enables I3D to effectively extract spatiotemporal features from videos and is widely used in tasks such as action recognition.

[0009] Prototype Feature Learning Transformer with Causal Attention: This model incorporates a causal attention mechanism to learn prototype features for each action category. Causal attention avoids leakage of future information by focusing on the portion of the input sequence relevant to the current time step, thereby improving the model's prediction accuracy. In action recognition, prototype features represent the typical characteristics of each action category, and the Transformer is used to learn representations of these features.

[0010] Causal Attention-Based Behavior Feature Enhancement Encoder: This encoder uses a causal attention mechanism to extract behavioral features from videos without future information. By focusing on the current and past frames, the model can capture the dynamic changes of actions and enhance the understanding of behavior.

[0011] Cosine similarity: Cosine similarity is a metric that measures the similarity between two vectors. It is calculated by dividing the dot product of the two vectors by the product of their moduli. In feature matching, cosine similarity is used to evaluate the similarity between video features and action category prototype features, helping to determine the most suitable prototype features.

[0012] Multilayer Perceptron (MLP): A multilayer perceptron is a feedforward neural network consisting of multiple fully connected layers. In feature fusion, MLP is used to fuse the matched prototype features with video features to generate video features with enhanced behavioral features.

[0013] Behavioral Feature Enhancement Decoder Based on Causal Attention Mechanism: This decoder uses the causal attention mechanism to decode the fused features into the final video feature representation. By focusing on historical information, the decoder is able to generate more accurate feature representations.

[0014] Fully connected layer: A fully connected layer is a type of layer in a neural network where every input node is connected to every output node. In action classification tasks, fully connected layers are used to map the final video features to the action category space and output the predicted probability for each category.

[0015] The technical solutions of the present invention are as follows: A first-person perspective online action segmentation method based on uncertainty includes the following steps: Step 1: Get the video that needs to be segmented online; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Input the features extracted in step 2 into the trained causal encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 4: Input the features extracted by the causal encoder in step 3 into the trained action prediction decoder based on the causal attention mechanism. Based on the video features extracted by the causal encoder module in step 3, the action prediction decoder is used to predict future action features. Step 5: Input the video features extracted by the causal encoder in step 3 and the future action features predicted by the action prediction decoder in step 4 into the trained context aggregation decoder based on the causal attention mechanism. Use the multi-layer perceptron to fuse the video features extracted in step 3 and the future action features predicted in step 4 to obtain video features containing context information. Step 6: Input the features containing context information fused in step 5 into the context aggregation decoder, use the decoder to obtain the final video features, input a fully connected layer, and obtain the action classification result of the context aggregation decoder; Step 7: Based on the probability distribution of the action classification results in step 6, use the evidence deep learning theory to calculate the uncertainty, and calculate the uncertainty of the prediction based on the uncertainty theory; Step 8: Based on the uncertainty value obtained in step 7, modify the context scenario results based on uncertainty, smooth the action predictions with high uncertainty, and make comprehensive modifications based on the predictions of the previous few frames of the action to obtain the final classification result.

[0016] The present invention also relates to an uncertainty-based first-person perspective online action segmentation system, specifically comprising: Acquisition unit: acquires the video that needs to be segmented online; I3D feature extractor: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1; Causal Encoder: The features extracted by the I3D feature extractor are input into the trained causal encoder based on the causal attention mechanism, and the causal encoder module is used to extract video features without future information; Action Prediction Decoder: The features extracted by the causal encoder are input into the trained action prediction decoder based on the causal attention mechanism. Based on the video features extracted by the causal encoder module, the action prediction decoder is used to predict future action features. Context Aggregation Decoder: The video features extracted by the causal encoder and the future action features predicted by the action prediction decoder are input into the trained context aggregation decoder based on the causal attention mechanism; Multi-layer perceptron: Use a multi-layer perceptron to fuse the extracted video features and predicted future action features to obtain video features containing contextual information; Action classification unit: The fused features containing context information are input into the context aggregation decoder, the decoder is used to obtain the final video features, and then input into a fully connected layer to obtain the action classification results of the context aggregation decoder; Uncertainty calculation unit: Based on the probability distribution of action classification results, uncertainty is calculated using evidence deep learning theory, and the uncertainty of the prediction is calculated based on uncertainty theory; Classification unit: Based on the uncertainty value obtained, the context scenario result is modified based on the uncertainty, the action prediction with high uncertainty is smoothed, and the prediction of the previous few frames of the action is combined for comprehensive modification to obtain the final classification result.

[0017] The beneficial effects of the present invention are as follows: This paper proposes an uncertainty-based first-person online action segmentation method for first-person online action segmentation. This method effectively addresses the issues of visual ambiguity and lack of subsequent context that often plague this task. This method enables the model to better address the challenges of action recognition in dynamic environments and complex scenes when processing first-person video streams.

[0018] This paper designs a context-aggregating transformer that combines autoregressive and autoencoding techniques to capture temporal dependencies in video streams, significantly enhancing the effectiveness of temporal contextualization and enabling better learning of feature representations for specific actions. This design improves the model's ability to understand action sequences and changing trends.

[0019] This paper proposes an evidence-based action contextualization method that quantifies the uncertainty in model decisions and dynamically adjusts the training and inference processes to mitigate the impact of blurred visual content. This enhances the model's robustness and accuracy when handling ambiguous action frames and uncertain scenes.

[0020] The online action segmentation method of this invention processes each frame of video data in real time, without waiting for future frames. This makes it well-suited for applications requiring high real-time performance, such as intelligent assistants and autonomous driving. Extensive experimental results demonstrate that compared to existing methods, this method performs superiorly in online action segmentation tasks, particularly in handling ambiguous actions and action boundaries, achieving higher accuracy and stability, resulting in significant performance improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments, wherein: Figure 1 Flowchart for the implementation of the present invention; Figure 2 This is a schematic diagram of the framework of the uncertainty-based first-person perspective online action segmentation method proposed in the present invention; Figure 3 This figure shows the effect of the present invention on the EgoPER dataset. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0023] I3D Feature Extractor: I3D (Inflated 3D ConvNet) is a deep learning model for video analysis. It simultaneously captures spatial and temporal features of videos by extending the convolution and pooling kernels of a 2D convolutional neural network (CNN) to 3D. This approach enables I3D to effectively extract spatiotemporal features from videos and is widely used in tasks such as action recognition.

[0024] Prototype Feature Learning Transformer with Causal Attention: This model incorporates a causal attention mechanism to learn prototype features for each action category. Causal attention avoids leakage of future information by focusing on the portion of the input sequence relevant to the current time step, thereby improving the model's prediction accuracy. In action recognition, prototype features represent the typical characteristics of each action category, and the Transformer is used to learn representations of these features.

[0025] Causal Attention-Based Behavior Feature Enhancement Encoder: This encoder uses a causal attention mechanism to extract behavioral features from videos without future information. By focusing on the current and past frames, the model can capture the dynamic changes of actions and enhance the understanding of behavior.

[0026] Cosine similarity: Cosine similarity is a metric that measures the similarity between two vectors. It is calculated by dividing the dot product of the two vectors by the product of their moduli. In feature matching, cosine similarity is used to evaluate the similarity between video features and action category prototype features, helping to determine the most suitable prototype features.

[0027] Multilayer Perceptron (MLP): A multilayer perceptron is a feedforward neural network consisting of multiple fully connected layers. In feature fusion, MLP is used to fuse the matched prototype features with video features to generate video features with enhanced behavioral features.

[0028] Behavioral Feature Enhancement Decoder Based on Causal Attention Mechanism: This decoder uses the causal attention mechanism to decode the fused features into the final video feature representation. By focusing on historical information, the decoder is able to generate more accurate feature representations.

[0029] Fully connected layer: A fully connected layer is a type of layer in a neural network where every input node is connected to every output node. In action classification tasks, fully connected layers are used to map the final video features to the action category space and output the predicted probability for each category.

[0030] Evidential Deep Learning Theory Calculates Uncertainty: Evidential Deep Learning is a method that uses evidence theory to quantify classification uncertainty. Unlike traditional softmax methods that output probabilities, Evidential Deep Learning uses a neural network to output "subjective opinions" and parameterizes them as a Dirichlet distribution, providing estimates of the predicted category and its uncertainty. This approach enables the model to explicitly express its confidence in the face of uncertainty.

[0031] SGD Optimizer: SGD (Stochastic Gradient Descent) is a commonly used optimization algorithm for training deep learning models. Unlike traditional batch gradient descent, SGD uses a mini-batch of data at a time to calculate gradients and update model parameters, which can speed up the training process and reduce memory consumption. The basic idea of SGD is that in each iteration, a mini-batch of data is randomly selected, its gradient is calculated, and the model parameters are updated based on this gradient.

[0032] like Figure 1 As shown, the implementation of the present invention can be divided into two stages, namely, model training and online action segmentation using the model.

[0033] Example 1 Step 1: Select a training dataset; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in each video in the training dataset in step 1.

[0034] Step 3: Construct a causal encoder based on the causal attention mechanism. For the video features in the training dataset in step 2, use the causal encoder module to extract video features without future information.

[0035] Step 4: Construct an action prediction decoder based on the causal attention mechanism. According to the video features extracted by the causal encoder module in step 3, the action prediction decoder is used to predict future action features.

[0036] Step 5: Input the future action features obtained in step 4 into the classification fully connected layer of the action prediction decoder to obtain the predicted action classification result.

[0037] Step 6: Build a contextual aggregation decoder based on the causal attention mechanism. This uses a multi-layer perceptron to fuse the video features extracted in Step 3 with the future action features predicted in Step 4 to obtain video features containing contextual information. The predicted future features complement the contextual information, making up for the lack of future information due to online conditions.

[0038] Step 7: Input the features fused in step 6 into the context aggregation decoder, use the decoder to obtain the final video features, input a fully connected layer, and obtain the action classification result of the context aggregation decoder.

[0039] Step 8: Based on the probability distribution of the action classification results in step 7, use the evidence deep learning theory to calculate the uncertainty, dynamically adjust the evidence learning loss weight according to the uncertainty, and calculate the classification loss based on the uncertainty.

[0040] Step 9: Based on the uncertainty value obtained in step 8, modify the context scenario results based on uncertainty, smooth the action predictions with high uncertainty, and make comprehensive modifications based on the predictions of the previous frames of the action.

[0041] Step 10: For the action prediction decoder in step 4, calculate the consistency loss based on the predicted action features in step 4 and the action features decoded by the context decoder in step 6. Calculate the action predictor prediction loss based on the predicted action classification results in step 5.

[0042] Step 11: Use the stochastic gradient descent SGD optimizer to optimize the model parameters, adjust the corresponding hyperparameters according to the different training data sets, and perform model training.

[0043] Step 12: Input the video into the trained uncertainty-based action segmentation network to verify the online action segmentation effect of the trained uncertainty-based online action segmentation method.

[0044] like Figure 2As shown in the figure, the core idea of the present invention is to adopt an uncertainty-based strategy for the online action segmentation task in the first-person perspective to enhance the robustness of the model in the face of uncertain actions and the accurate understanding of the current action characteristics, and reduce the excessive dependence on prior contextual information.

[0045] In our context-aggregating transformer module, we combine autoregressive and autoencoding designs to capture temporal dependencies in video streams. Leveraging prior knowledge and the learning capabilities of neural networks, we extract information from historical frames that may be useful for current action judgment. This module significantly enhances the effectiveness of temporal contextualization, enabling better learning of feature representations for specific actions and overcoming the limitations of streaming video due to the lack of subsequent context.

[0046] In our evidence-based action contextualization module, we mitigate the impact of ambiguous visual content by quantifying the uncertainty in model decisions and dynamically adjusting the training and inference processes. Specifically, during training, we incorporate uncertainty theory to prioritize difficult-to-predict actions with high uncertainty, allowing the model to better understand ambiguous motion. During inference, we employ a smoothing strategy for highly uncertain predictions, combining predictions from previous frames to make comprehensive modifications. This dynamic, uncertainty-based adjustment strategy enhances the model's robustness and accuracy when handling ambiguous motion frames and uncertain scenes.

[0047] It should be emphasized that the method described in the present invention ensures the consistency of the training process and the inference process, adopts a strategy of dynamically changing the network according to uncertainty, and realizes efficient and accurate online action segmentation in the absence of future information, which greatly improves the action segmentation effect under the limitation of online inspection.

[0048] like Figure 3 As shown, the present invention proposes an uncertainty-based first-person perspective online action segmentation method, which adopts a strategy of dynamically changing the network according to uncertainty to achieve efficient and accurate online action segmentation in the absence of future information. Compared with other methods, it improves the accuracy and reduces the problem of over-segmentation.

[0049] Example 2 An uncertainty-based first-person online action segmentation method, combined with Figure 2 As a preferred technical solution, step 2 includes in more detail: using the I3D feature extractor to extract the frame-level video features .

[0050] As a preferred technical solution, step 3 includes in more detail: using a causal mask to shield future information to ensure that the training process and reasoning process of the first-person perspective online action segmentation are consistent. The formula of the attention mechanism is expressed as: in represents the causal mask, Represent the query vector, key vector and value vector in the attention mechanism respectively.

[0051] As a preferred technical solution, step 4 includes in more detail: using the action prediction decoder to predict the future action features, and obtaining the predicted action features. , the formula is as follows: in represents the action prediction decoder based on causal attention mechanism, represents the video features extracted by the causal encoder in step 3.

[0052] As a preferred technical solution, step 6 includes in more detail: using a multi-layer perceptron to fuse the video features extracted in step 3 and the future action features predicted in step 4 to obtain video features containing context information .

[0053] As a preferred technical solution, step 7 includes in more detail: inputting the features fused in step 6 into the context aggregation decoder, and using the decoder to obtain the final video features. It can be expressed as follows: in represents the action prediction decoder based on causal attention mechanism, Represents the parameters of the causal attention-based action prediction decoder.

[0054] As a preferred technical solution, step 8 includes in more detail: according to the probability distribution of the action classification results in step 7, the uncertainty is calculated using the evidence deep learning theory to obtain the matching evidence , based on matching evidence Calculate the uncertainty: Quantify the uncertainty of each action classification result to provide a more reliable and robust basis for subsequent action segmentation. Then dynamically adjust the weight of the evidence learning loss according to the uncertainty and calculate the uncertainty-based classification loss, which is expressed as follows: in represents the dynamic adjustment weight based on uncertainty, represents the uncertainty at time t, represents the probability of being predicted as the background class at time t, It represents the probability of predicting action i at time t.

[0055] As a preferred technical solution, step 9 includes, in more detail, modifying the contextual scenario results based on the uncertainty value obtained in step 8, smoothing the action predictions with high uncertainty, and performing comprehensive modification based on the predictions of the previous frames of the action. This can be expressed as follows: As a preferred technical solution, step 10 includes, in more detail, calculating a consistency loss for the action prediction decoder in step 4 based on the predicted action features in step 4 and the action features decoded by the context decoder in step 6, to constrain the predicted action features. Specifically, it is desirable that the predicted action features have the same offset as the features input to the context aggregation decoder in step 7. The smaller the difference in features, the better the prediction effect. Therefore, the consistency loss formula is set as follows: in represents the video features at time t, express Future prediction features of the moment.

[0056] According to the predicted action classification results in step 5, the action predictor prediction loss is calculated to constrain the action prediction decoder prediction results. The formula is as follows: As a preferred technical solution, step 11 includes in more detail: in order to reduce the over-segmentation problem in action segmentation, a smoothing loss is introduced, which is expressed by the following formula: Finally, combining the uncertainty-based classification loss, smoothness loss, consistency loss, and action predictor prediction loss, the total loss of the uncertainty-based first-view online action segmentation method is obtained as follows: in They are the corresponding weights of smoothing loss, consistency loss, and action predictor prediction loss.

[0057] As a preferred technical solution, step 12 includes in more detail: using the trained model to perform first-person online action segmentation, obtaining the action classification corresponding to each frame, obtaining the action segmentation result, and then calculating the accuracy, edit distance and F1 score respectively.

[0058] Optionally, to test the model's performance in the first-person online action segmentation task, the present invention employs multiple evaluation metrics for this task, such as Acc, Edit Score, and F1 Score. Acc measures the classification performance of each frame by calculating the proportion of correctly classified frames out of the total number of frames. This metric reflects the model's ability to grasp action details. Edit Score assesses the similarity between the predicted and true sequences by determining the minimum number of operations (insertion, deletion, or substitution) required to align the two. The Edit Distance reflects the temporal coherence of the predicted segmentation; a smaller Edit Distance indicates a closer temporal and structural alignment between the predicted and true sequences, reflecting the logicality and coherence of the model's segmentation. The F1 Score is the harmonic mean of the precision and recall of action segments at 10%, 25%, and 50% overlap thresholds based on the Intersection-Over-Union (IoU) criterion. IoU determines whether the predicted and true segments have sufficient spatial overlap to be considered a match. This evaluation method ensures that the accuracy of the segmentation boundaries is taken into account, while measuring the matching effect under different degrees of overlap. It also punishes over-segmentation by aligning the predicted segments with the real segments, thereby comprehensively evaluating the action segmentation effect of the model under different overlap requirements.

[0059] After the training is completed, the first-person perspective online action segmentation method of the present invention includes the following steps: Step 1: Get the video that needs to be segmented online; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Input the features extracted in step 2 into the trained causal encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 4: Input the features extracted by the causal encoder in step 3 into the trained action prediction decoder based on the causal attention mechanism. Based on the video features extracted by the causal encoder module in step 3, the action prediction decoder is used to predict future action features. Step 5: Input the video features extracted by the causal encoder in step 3 and the future action features predicted by the action prediction decoder in step 4 into the trained context aggregation decoder based on the causal attention mechanism. Use the multi-layer perceptron to fuse the video features extracted in step 3 and the future action features predicted in step 4 to obtain video features containing context information. Step 6: Input the features containing context information fused in step 5 into the context aggregation decoder, use the decoder to obtain the final video features, input a fully connected layer, and obtain the action classification result of the context aggregation decoder; Step 7: Based on the probability distribution of the action classification results in step 6, use the evidence deep learning theory to calculate the uncertainty, and calculate the uncertainty of the prediction based on the uncertainty theory; Step 8: Based on the uncertainty value obtained in step 7, modify the context scenario results based on uncertainty, smooth the action predictions with high uncertainty, and make comprehensive modifications based on the predictions of the previous few frames of the action to obtain the final classification result.

[0060] Three scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention: The first scenario case study: The model is trained and tested on the GTEA dataset, and the results are compared with previous methods. The GTEA dataset contains 28 videos covering seven daily activities, such as making coffee and cheese sandwiches, performed by four participants. The videos were recorded with a GoPro camera with a resolution of 1280×720 pixels. The dataset provides frame-level annotations for 11 action categories, with an average of 20 action instances per video, a total of 31,222 frames, and a frame sampling rate of 15 fps. The GTEA dataset is divided into four subsets for model training, validation, and testing, and the reported performance is the average result obtained from these four subsets. The comparison of the online action segmentation effect is shown in Table 1.

[0061] Table 1 Comparison of online action segmentation effects on the GTEA dataset Second scenario: Model training and testing were performed on the EgoProceL dataset, and the results were compared with previous methods. The EgoProceL dataset is a large-scale first-person perspective resource, containing 62 hours of video recordings across 16 different tasks (average task duration of 13 minutes). It emphasizes identifying key procedural steps rather than exhaustive action coverage, making it suitable for research on human-computer interaction, action prediction, and cross-modal analysis. This dataset integrates existing resources such as CMU-MMAC, EGTEA Gaze+, MECCANO, and EPIC-Tents, and adds new tasks such as PC assembly / disassembly to increase diversity. The dataset consists of five different subsets of first-person perspective videos. The model was trained separately for each subset, and the final evaluation metric was the average of all subsets. A comparison of the results of online action segmentation is shown in Table 2.

[0062] Table 2 Comparison of online action segmentation effects on the EgoProceL dataset The third scenario uses the proposed method on the EgoPER dataset to demonstrate actual results for online action segmentation of untrimmed videos. The EgoPER dataset contains 386 untrimmed videos (213 normal and 173 error instances, totaling 28 hours of footage) focusing on five cooking-related tasks. Recorded using the Hololens 2, the average duration is 4.4 minutes. Table 3 compares the performance of online action segmentation.

[0063] Table 3 Comparison of online action segmentation effects on the EgoPER dataset The fourth scenario example: On the EgoPER dataset, the actual results of online action segmentation for a given video using the method proposed in this invention. Figure 3 As shown in the figure, for the given video data, in the case of "making pastry", our method improves the accuracy and reduces the over-segmentation problem. For example, in the actions of "inserting toothpicks" and "trimming the ends", our method improves the accuracy compared with previous methods. In the actions of "trimming the ends" and "cutting with dental floss", our method reduces over-segmentation compared with previous methods and improves the online action segmentation effect.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A first-person perspective online action segmentation method based on uncertainty, characterized by: The specific steps include: Step 1: Get the video that needs to be segmented online; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Input the features extracted in step 2 into the trained causal encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 4: Input the features extracted by the causal encoder in step 3 into the trained action prediction decoder based on the causal attention mechanism. Based on the video features extracted by the causal encoder module in step 3, the action prediction decoder is used to predict future action features. Step 5: Input the video features extracted by the causal encoder in step 3 and the future action features predicted by the action prediction decoder in step 4 into the trained context aggregation decoder based on the causal attention mechanism. Use the multi-layer perceptron to fuse the video features extracted in step 3 and the future action features predicted in step 4 to obtain video features containing context information. Step 6: Input the features containing context information fused in step 5 into the context aggregation decoder, use the decoder to obtain the final video features, input a fully connected layer, and obtain the action classification result of the context aggregation decoder; Step 7: Based on the probability distribution of the action classification results in step 6, use the evidence deep learning theory to calculate the uncertainty, and calculate the uncertainty of the prediction based on the uncertainty theory; Step 8: Based on the uncertainty value obtained in step 7, modify the context scenario results based on uncertainty, smooth the action predictions with high uncertainty, and make comprehensive modifications based on the predictions of the previous few frames of the action to obtain the final classification result.

2. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 2 includes more details: using I3D feature extractor to extract frame-level video features .

3. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 3 includes more details: using causal masks to block future information to ensure that the training process and inference process of first-person online action segmentation are consistent. The formula of the attention mechanism is expressed as: in represents the causal mask, Represent the query vector, key vector and value vector in the attention mechanism respectively.

4. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 4 includes more details: using the action prediction decoder to predict the future action features and obtain the predicted action features , the formula is as follows: in represents the action prediction decoder based on causal attention mechanism, represents the video features extracted by the causal encoder in step 3.

5. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 5 includes more details: using a multi-layer perceptron to fuse the video features extracted in step 3 and the future action features predicted in step 4 to obtain video features containing contextual information .

6. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 6 includes more details: inputting the features fused in step 6 into the context aggregation decoder, and using the decoder to obtain the final video features, which can be expressed as follows: in represents the action prediction decoder based on causal attention mechanism, Represents the parameters of the causal attention-based action prediction decoder.

7. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 7 includes in more detail: according to the probability distribution of the action classification results in step 6, the uncertainty is calculated using the evidence deep learning theory to obtain the matching evidence , based on matching evidence Calculate the uncertainty: Quantify the uncertainty of each action classification result, thereby providing a more reliable and robust basis for subsequent action segmentation.

8. The uncertainty-based first-person perspective online action segmentation method according to claim 1, characterized in that: Step 8 includes, in more detail, modifying the contextual scenario results based on the uncertainty value obtained in step 7, smoothing the action predictions with high uncertainty, and performing comprehensive modification based on the predictions of the previous frames of the action, which can be expressed as the following formula: 。 9. A first-person perspective online action segmentation system based on uncertainty, characterized by: Specifically include: Acquisition unit: acquires the video that needs to be segmented online; I3D feature extractor: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1; Causal Encoder: The features extracted by the I3D feature extractor are input into the trained causal encoder based on the causal attention mechanism, and the causal encoder module is used to extract video features without future information; Action Prediction Decoder: The features extracted by the causal encoder are input into the trained action prediction decoder based on the causal attention mechanism. Based on the video features extracted by the causal encoder module, the action prediction decoder is used to predict future action features. Context Aggregation Decoder: The video features extracted by the causal encoder and the future action features predicted by the action prediction decoder are input into the trained context aggregation decoder based on the causal attention mechanism; Multi-layer perceptron: Use a multi-layer perceptron to fuse the extracted video features and predicted future action features to obtain video features containing contextual information; Action classification unit: The fused features containing context information are input into the context aggregation decoder, the decoder is used to obtain the final video features, and then input into a fully connected layer to obtain the action classification results of the context aggregation decoder; Uncertainty calculation unit: Based on the probability distribution of action classification results, uncertainty is calculated using evidence deep learning theory, and the uncertainty of the prediction is calculated based on uncertainty theory; Classification unit: Based on the uncertainty value obtained, the context scenario result is modified based on the uncertainty, the action prediction with high uncertainty is smoothed, and the prediction of the previous few frames of the action is combined for comprehensive modification to obtain the final classification result.

Images