Behavior-centered first-view-angle online action segmentation method

Abstract

The invention discloses a behavior-centered first-view-angle online action segmentation method, relates to the technical field of video action analysis, and aims to perform behavior-centered feature enhancement by using a prototype and realize efficient and accurate online action segmentation under the condition of information loss in the future. The invention adopts a behavior center feature enhancement method. The method comprises two key parts: 1, behavior characteristics are learned by using a prototype, and the generality of behaviors in different environments is captured through the prototype, so that the capability of capturing shared characteristics of the same behaviors in different environments is enhanced; 2, behavior feature enhancement: feature enhancement is carried out by using intrinsic characteristics of a current behavior and by virtue of common sense captured by a prototype; in this way, dependence on context information is reduced, and the problems existing in the prior art can be effectively solved. The method can be widely applied to scenes related to first person view angle video action analysis, such as human body behavior monitoring based on wearable equipment, character action understanding in augmented reality and virtual reality scenes and the like.

Description

Technical Field

[0001] The present invention belongs to the field of first-perspective online action segmentation in action segmentation, and particularly relates to online recognition and segmentation of continuous actions in a video stream. Background Art

[0002] In today's society, the prevalence of wearable cameras and first-person perspective technology has become a trend. They have greatly enriched our visual experience and provided a broad space for the application of augmented reality and virtual reality technologies. As these technologies become increasingly mature and popular, they have seamlessly integrated into our daily lives, from gaming and entertainment to education and training, from healthcare to industrial design, and are almost ubiquitous.

[0003] This convergence of technologies is not only changing how we interact with the digital world but also driving a pressing need for in-depth understanding and analysis of unedited first-person video data. This demand is particularly pronounced in the field of action recognition and analysis. First-person video data often contains rich action information and context, which is crucial for understanding user intent, behavioral patterns, and real-time interactions. For example, in intelligent surveillance systems, analyzing first-person video data can identify abnormal behavior in real time, provide timely warnings, and safeguard public safety. In virtual reality games, capturing player movements can enable more natural and realistic interactive experiences. However, unlike traditional offline action segmentation tasks, online first-person action segmentation faces more stringent challenges. It requires models to process every frame of the video stream in real time, strictly along the temporal dimension, and without access to future frames. This real-time nature necessitates models that can quickly and accurately identify and segment continuous action segments, which is crucial for applications such as natural human-computer interaction, intelligent surveillance, and virtual reality experiences.

[0004] In online action segmentation scenarios, the model must rely on current observations and previous contextual information to interpret each frame in real time. This real-time nature requires the model to possess not only efficient computational capabilities but also powerful feature extraction and decision-making capabilities. However, due to the specific nature of online action segmentation, traditional action segmentation methods face two major challenges: First, due to the lack of access to future frame information, the model is limited in its understanding of action semantics and struggles to capture long-range dependencies; second, over-reliance on previous contextual information can lead to insufficient feature representation of the current action, especially at action blur or action boundaries.

[0005] To overcome these challenges, researchers have begun focusing on how to effectively capture the essential characteristics of current behavior in online action segmentation tasks and exploit the commonalities and similarities of actions in first-person video data. This requires models to not only process real-time visual input, but also adapt to changes in actions in different environments and accurately identify and segment individual action fragments. Researchers are exploring various approaches, including introducing attention mechanisms, using recurrent neural networks or long short-term memory networks to capture temporal dependencies, and employing transfer learning and domain adaptation techniques to improve the model's generalization capabilities.

[0006] In summary, online action segmentation, as an emerging research field, combines the latest advances in computer vision, machine learning, and artificial intelligence to improve the understanding and analysis of first-person perspective video data. With the continuous advancement of technology and growing application demand, we expect to see more innovations and breakthroughs in this field, providing users with more intelligent and personalized interactive experiences. In the future, we can expect to see widespread application of online action segmentation technology in a variety of fields, including smart assistants, health monitoring, and autonomous driving. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing first-person perspective online action segmentation methods, construct a behavior-centric feature enhancement network structure, and perform behavior-centric feature enhancement by combining prototype technology and long short-term memory to achieve efficient and accurate online action segmentation in the absence of future information.

[0008] The technical solutions of the present invention are as follows: Step 1: Select the video to be segmented online for input; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Learn the transformer based on the trained prototype features based on the causal attention mechanism to obtain the prototype features of each type of action.

[0009] Step 4: Input the features extracted in step 2 into the trained behavioral feature enhancement encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 5: Calculate the cosine similarity between the video features in step 4 and the prototype features of each action category in step 3; obtain the prototype features that best match the video features based on the cosine similarity with the prototype, and obtain the common features of behaviors in different environments; Step 6: Use a multi-layer perceptron to fuse the prototype features matched in step 5 and the video features in step 4 to obtain video features with enhanced behavioral features; Step 7: Input the video features with enhanced behavioral features obtained in step 6 into the behavioral feature enhancement decoder based on the causal attention mechanism to obtain the final video features, and input them into a fully connected layer to obtain the first-person perspective online action classification results.

[0010] The present invention also relates to a behavior-centric first-person perspective online action segmentation system, specifically comprising: Acquisition unit: input the video that needs to be segmented online; I3D feature extractor: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video; Prototype feature learning transformer: Based on the trained prototype feature learning transformer based on the causal attention mechanism, the prototype features of each type of action are obtained; Behavior Feature Enhancement Encoder: The features extracted by the I3D feature extractor are input into the trained behavior feature enhancement encoder based on the causal attention mechanism, and the causal encoder module is used to extract video features without future information; Cosine similarity calculation unit: Calculates the cosine similarity between the video features without future information and the prototype features of each action category; obtains the prototype features that best match the video features based on the cosine similarity with the prototype, and obtains the common characteristics of behaviors in different environments; Multi-layer perceptron: Use a multi-layer perceptron to fuse the matched prototype features and video features to obtain video features with enhanced behavioral features; Behavior Feature Enhancement Decoder: The obtained video features with enhanced behavior features are input into the behavior feature enhancement decoder based on the causal attention mechanism to obtain the final video features, which are then input into a fully connected layer to obtain the first-person perspective online action classification results.

[0011] The beneficial effects of the present invention are as follows: This paper proposes a behavior-centric, first-person perspective online action segmentation method. By emphasizing the commonalities and inherent characteristics of current behaviors, it significantly improves feature representation capabilities. This approach enables the model to more accurately capture and understand user behavioral intentions when processing first-person perspective video data.

[0012] This paper introduces a prototype technique to learn behavioral commonalities. This allows prototypes to capture common sense about current behavior in different environments, enhancing the model's ability to interpret blurred action frames in first-person perspective videos. This learning strategy improves the model's adaptability to action boundaries and variations.

[0013] This paper proposes a behavioral feature enhancement strategy that effectively reduces the model's reliance on contextual information by integrating the intrinsic characteristics of the current behavior with behavioral common sense extracted through behavioral prototype feature learning. This not only improves the model's real-time performance but also enhances its robustness in dynamic and complex environments.

[0014] The online action segmentation method of the present invention processes each frame of video data in real time, without waiting for future frames. This makes the model well-suited for applications requiring immediate feedback, such as virtual reality interaction and intelligent surveillance. Experimental results demonstrate that the present invention achieves significant performance improvements in online action segmentation tasks compared to existing methods. In particular, the method demonstrates enhanced accuracy and reliability when processing fuzzy actions and action boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments, wherein: Figure 1 Flowchart for the implementation of the present invention; Figure 2 This is a schematic diagram of the framework of the behavior-centric first-person perspective online action segmentation method proposed in the present invention; Figure 3 This figure shows the effect of the present invention on the EgoPER dataset. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0017] I3D Feature Extractor: I3D (Inflated 3D ConvNet) is a deep learning model for video analysis. It simultaneously captures spatial and temporal features of videos by extending the convolution and pooling kernels of a 2D convolutional neural network (CNN) to 3D. This approach enables I3D to effectively extract spatiotemporal features from videos and is widely used in tasks such as action recognition.

[0018] Prototype Feature Learning Transformer with Causal Attention: This model incorporates a causal attention mechanism to learn prototype features for each action category. Causal attention avoids leakage of future information by focusing on the portion of the input sequence relevant to the current time step, thereby improving the model's prediction accuracy. In action recognition, prototype features represent the typical characteristics of each action category, and the Transformer is used to learn representations of these features.

[0019] Causal Attention-Based Behavior Feature Enhancement Encoder: This encoder uses a causal attention mechanism to extract behavioral features from videos without future information. By focusing on the current and past frames, the model can capture the dynamic changes of actions and enhance the understanding of behavior.

[0020] Cosine similarity: Cosine similarity is a metric that measures the similarity between two vectors. It is calculated by dividing the dot product of the two vectors by the product of their moduli. In feature matching, cosine similarity is used to evaluate the similarity between video features and action category prototype features, helping to determine the most suitable prototype features.

[0021] Multilayer Perceptron (MLP): A multilayer perceptron is a feedforward neural network consisting of multiple fully connected layers. In feature fusion, MLP is used to fuse the matched prototype features with video features to generate video features with enhanced behavioral features.

[0022] Behavioral Feature Enhancement Decoder Based on Causal Attention Mechanism: This decoder uses the causal attention mechanism to decode the fused features into the final video feature representation. By focusing on historical information, the decoder is able to generate more accurate feature representations.

[0023] Fully connected layer: A fully connected layer is a type of layer in a neural network where every input node is connected to every output node. In action classification tasks, fully connected layers are used to map the final video features to the action category space and output the predicted probability for each category.

[0024] like Figure 1 As shown, the implementation of the present invention can be divided into two stages, namely, model training and online action segmentation using the model.

[0025] Example 1 Step 1: Select a training dataset; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in each video in the training dataset in step 1.

[0026] Step 3: Construct a prototype feature learning encoder based on the causal attention mechanism. For the video features in the training dataset in step 2, use the causal encoder module to extract video features without future information.

[0027] Step 4: Based on the video features extracted by the prototype feature learning encoder module in step 3, use prototype technology to cluster each type of action to obtain the prototype features of each type of action, capture the commonalities of behaviors in different environments, and enhance the ability to capture shared features of the same behaviors in different environments.

[0028] Step 5: Construct a prototype feature learning decoder based on the causal attention mechanism. Use the causal decoder to classify the video features extracted by the prototype feature learning encoder module in step 3 to obtain the action classification results of the network, which are used to optimize the parameters of the prototype learning network.

[0029] Step 6: Introduce the action category balancing module to balance the weights to solve the imbalance problem between action categories in the first-person video and strengthen the learning of difficult-to-recognize frames and relatively few categories.

[0030] Step 7: Construct a behavioral feature enhancement encoder based on the causal attention mechanism. For the video features in the training dataset in step 2, use the causal encoder module to extract video features without future information.

[0031] Step 8: Calculate the cosine similarity between the video features in step 7 and the prototype features of each action category in step 4; obtain the prototype features that best match the video features based on the cosine similarity with the prototype, and obtain the common features of behaviors in different environments.

[0032] Step 9: Use a multi-layer perceptron to fuse the prototype features matched in step 8 and the video features in step 7 to obtain video features with enhanced behavioral features.

[0033] Step 10: For the video features enhanced by the behavioral features in step 9, a short-term information enhancement mask is used to capture the short-term features of the video and strengthen the focus on the intrinsic characteristics of the current behavior itself.

[0034] Step 11: Construct a behavioral feature enhancement decoder based on the causal attention mechanism, input the long-term and short-term video features selected in steps 9 and 10 into the decoder in parallel to obtain the final video features, and input them into a fully connected layer to obtain the classification results.

[0035] Step 12: Use the stochastic gradient descent SGD optimizer to optimize the model parameters, adjust the corresponding hyperparameters according to the different training data sets, and perform model training.

[0036] Step 13: Input the video into the trained behavior-centric action segmentation network to verify the online action segmentation effect of the trained behavior-centric action segmentation network.

[0037] like Figure 2 As shown, the core idea of this invention is to use a behavior-centric feature enhancement strategy for untrimmed video data from a first-person perspective to enhance the feature representation of the current behavior and reduce dependence on contextual information. This method mainly includes two key components: behavior prototype learning and behavior feature enhancement.

[0038] In the behavior prototype learning module, we first extract frame-level features in the prototype learning network using a causal encoder. These frame-level features are then clustered to capture shared characteristics of behaviors across different environments. These clustered prototypes are then matched with features in the presented behavior enhancement module to provide commonsense understanding of the currently presented behavior, thereby improving its ability to interpret ambiguous action frames from a first-person perspective.

[0039] In the behavior feature enhancement module, we combine the intrinsic characteristics of the currently presented behavior with the commonsense features captured by behavior prototype learning for feature enhancement. Specifically, the frame-level features extracted by the causal encoder are matched and fused with the prototype representation. These enhanced features are then passed to the behavior enhancement decoder, where attention masks and a dual parallel structure are used to strengthen the action-level understanding of the currently presented behavior, minimizing reliance on contextual information.

[0040] It should be emphasized that the method described in the present invention ensures the consistency of the training process and the inference process, and uses prototype learning to capture the commonalities of action presentation and the idea of long-term and short-term learning to focus on the characteristics of the behavior itself, reducing dependence on contextual information and greatly improving the action segmentation effect under the limitation of online inspection.

[0041] like Figure 3 As shown in Figure 2, this paper proposes a behavior-centric, first-person perspective online action segmentation method that uses prototypes for behavior-centric feature enhancement, achieving efficient and accurate online action segmentation in the absence of future information. Compared to other methods, this method improves accuracy and reduces the problem of over-segmentation.

[0042] Example 2 A behavior-centric first-person online action segmentation method, combining Figure 2 As a preferred technical solution, step 2 includes in more detail: using the I3D feature extractor to extract the frame-level video features .

[0043] As a preferred technical solution, step 3 includes in more detail: using a causal mask to shield future information to ensure that the training process and reasoning process of the first-person perspective online action segmentation are consistent. The formula of the attention mechanism is expressed as: in Represents an encoder for prototypical feature learning based on causal attention mechanism. represents the causal mask, Represent the query vector, key vector and value vector in the attention mechanism respectively.

[0044] As a preferred technical solution, step 4 includes in more detail: clustering the video features using K-means to obtain prototype features: , where i represents the action type and k represents the number of prototype cluster centers. As a preferred technical solution, step 6 includes the following in more detail: in order to make the model pay more attention to low-confidence prediction results during training, especially those belonging to the minority class, thereby effectively alleviating the class imbalance problem caused by the different durations of different actions and improving the accuracy of the model in segmenting various actions in the first-person perspective video. The following formula is introduced: in Indicates dynamic adjustment of balance weight, Represents the total number of frames of all actions, Represents the number of action types, Indicates the total number of frames corresponding to action i. represents the action label corresponding to time t, represents the action prediction corresponding to time t.

[0045] As a preferred technical solution, step 8 includes, in more detail: obtaining the prototype features that best match the video features based on the cosine similarity between the video features extracted in step 7 and the prototype features, thereby obtaining the common features of behaviors in different environments. This can be expressed as follows: in represents the video features extracted in step 4, Represents a prototype feature.

[0046] As a preferred technical solution, step 9 includes in more detail: fusing the video features extracted in step 7 and the prototype features extracted in step 8 using a multi-layer perceptron to obtain video features with enhanced behavioral features, which can be expressed by the following formula: As a preferred technical solution, step 10 includes the following in more detail: For the video features extracted in step 9 with enhanced behavioral features, we not only hope to establish long-term dependency modeling, but also hope that the model will pay more attention to the intrinsic characteristics of the action behavior. This is to address the problem that the lack of subsequent context under online constraints limits the effectiveness of prior context for learning action semantics. Therefore, short-term behavior-related frames are selected for feature enhancement, which can be expressed as follows: in represents the behavioral feature enhanced decoder based on causal attention mechanism, Indicates the short-term information enhancement mask occlusion ratio.

[0047] As a preferred technical solution, step 11 includes, in more detail, the following: The long-term and short-term video features selected in steps 9 and 10 are fed into the decoder in parallel, where the features are enhanced by utilizing both the intrinsic characteristics of the currently presented behavior and the behavioral commonalities captured by the prototype technology. The final video features are then fed into a fully connected layer to obtain the classification results, which are then fed into the behavior-centric action segmentation network. This can be expressed as follows: As a preferred technical solution, step 12 includes in more detail: in order to reduce the over-segmentation problem in action segmentation, a smoothing loss is introduced, which can be expressed by the following formula: Finally, by combining the smoothing loss, behavior enhancement loss, and balanced category loss, the total loss of the behavior-centered first-perspective online action segmentation method is obtained as follows: in They are the corresponding weights of smoothing loss, behavior enhancement loss, and balanced category loss.

[0048] As a preferred technical solution, step 13 includes in more detail: using the trained model to perform first-person online action segmentation, obtaining the action classification corresponding to each frame, obtaining the action segmentation result, and then calculating the accuracy, edit distance and F1 score respectively.

[0049] After the training is completed, the actual steps of using the method of the present invention are as follows: Step 1: Select the video to be segmented online for input; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Learn the transformer based on the trained prototype features based on the causal attention mechanism to obtain the prototype features of each type of action.

[0050] Step 4: Input the features extracted in step 2 into the trained behavioral feature enhancement encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 5: Calculate the cosine similarity between the video features in step 4 and the prototype features of each action category in step 3; obtain the prototype features that best match the video features based on the cosine similarity with the prototype, and obtain the common features of behaviors in different environments; Step 6: Use a multi-layer perceptron to fuse the prototype features matched in step 5 and the video features in step 4 to obtain video features with enhanced behavioral features; Step 7: Input the video features with enhanced behavioral features obtained in step 6 into the behavioral feature enhancement decoder based on the causal attention mechanism to obtain the final video features, and input them into a fully connected layer to obtain the first-person perspective online action classification results.

[0051] Optionally, to test the model's online action segmentation performance, the present invention employs various evaluation metrics for the first-person online action segmentation task, including Acc, Eidt Score, and F1 Score. Acc measures per-frame classification performance by calculating the percentage of correctly classified frames out of the total number of frames, reflecting the model's grasp of action details. EditScore assesses the similarity between the predicted and true sequences by determining the minimum number of operations (insertion, deletion, or substitution) required to align them. Edit distance reflects the temporal coherence of the predicted segmentation; a smaller distance indicates closer temporal and structural proximity between the two sequences, reflecting the logicality and coherence of the model's segmentation. F1 Score calculates the harmonic mean of precision and recall for action segments at 10%, 25%, and 50% overlap thresholds based on the Intersection over Union (IoU) criterion. IoU determines whether the predicted and true segments overlap sufficiently to be considered a match. This evaluation method ensures that both the accuracy of the segmentation boundaries and the degree of overlap are taken into account, and penalizes over-segmentation by aligning the predicted segments with the true segments, thereby comprehensively evaluating the model's effect on action segmentation under different overlap requirements.

[0052] Three scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention: First scenario case: The model was trained and tested on the GTEA dataset, and its performance was compared with previous methods. The GTEA dataset contains 28 videos covering seven everyday activities, such as making coffee and cheese sandwiches. These videos provide frame-level annotations for 11 action categories, with an average of 20 action instances per video, totaling 31,222 frames. We partitioned the data into four subsets for model training, validation, and testing, respectively. The reported performance is the average of the results obtained from these four subsets. A comparison of the performance of online action segmentation is shown in Table 1.

[0053] Table 1 Comparison of online action segmentation effects on the GTEA dataset Second scenario example: Model training and testing are performed on the EgoProceL dataset, and the results are compared with previous methods. EgoProceL is a large-scale first-person perspective dataset containing 62 hours of recordings, involving 16 tasks, with an average duration of 13 minutes per task. It integrates resources such as CMU-MMAC, EGTEA Gaze+, MECCANO, and EPIC-Tents, and adds new tasks such as PC assembly to improve diversity. The dataset consists of first-person perspective videos from five different subsets. The model is trained for each subset separately, and the final evaluation metric is the average of all subsets. The comparison of the online action segmentation effect is shown in Table 2.

[0054] Table 2 Comparison of online action segmentation effects on the EgoProceL dataset The third scenario case: On the EgoPER dataset, the actual results of online action segmentation for a given untrimmed video using the method proposed in the present invention are shown. The EgoPER dataset covers 386 unedited videos, including 213 normal video instances and 173 erroneous instances, with a total length of 28 hours. This dataset mainly focuses on five cooking-related tasks. As shown in Table 3, for the videos of the five cooking-related tasks included in the EgoPER dataset, the method proposed in the present invention can accurately segment the actions regardless of whether the action boundaries are clear or blurred. In particular, because our proposed behavior-centric feature enhancement method can effectively utilize the intrinsic characteristics of the current behavior and the common features across environments, it has more advantages than other methods in dealing with complex multimodal tasks such as EgoPER.

[0055] The fourth scenario example: On the EgoPER dataset, the actual results of online action segmentation for a given video using the method proposed in this invention. Figure 3As shown in the figure, for the given video data, in the case of "making tea", our method improves the accuracy and reduces the over-segmentation problem. For example, in the action of "pouring water from the kettle into the cup", our method improves the accuracy compared with the previous method. In the action of "putting the tea bag into the cup", our method reduces over-segmentation compared with the previous method and improves the model effect.

[0056] Table 3 Comparison of online action segmentation effects on the EgoPER dataset The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A behavior-centric first-person perspective online action segmentation method, characterized by: The specific steps include: Step 1: Get the video that needs to be segmented online; Step 2: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video in step 1. Step 3: Learn the transformer based on the trained prototype features based on the causal attention mechanism to obtain the prototype features of each type of action; Step 4: Input the features extracted in step 2 into the trained behavioral feature enhancement encoder based on the causal attention mechanism, and use the causal encoder module to extract video features without future information; Step 5: Calculate the cosine similarity between the video features in step 4 and the prototype features of each action category in step 3; obtain the prototype features that best match the video features based on the cosine similarity with the prototype, and obtain the common features of behaviors in different environments; Step 6: Use a multi-layer perceptron to fuse the prototype features matched in step 5 and the video features in step 4 to obtain video features with enhanced behavioral features; Step 7: Input the video features with enhanced behavioral features obtained in step 6 into the behavioral feature enhancement decoder based on the causal attention mechanism to obtain the final video features, and input them into a fully connected layer to obtain the first-person perspective online action classification results.

2. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 2 includes more details: using I3D feature extractor to extract frame-level video features .

3. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 3 includes more details: using causal masks to block future information to ensure that the training process and inference process of first-person online action segmentation are consistent. The formula of the attention mechanism is expressed as: ;in represents the encoder for prototypical feature learning based on causal attention mechanism, represents the causal mask, Represent the query vector, key vector and value vector in the attention mechanism respectively.

4. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 4 includes more details: learning a transformer based on the trained prototype features based on the causal attention mechanism to obtain the prototype features: , where i represents the action type and k represents the number of prototype cluster centers.

5. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 5 includes, in more detail: obtaining the prototype features that best match the video features based on the cosine similarity between the video features extracted in step 4 and the prototype features, and obtaining the common features of behaviors in different environments, which can be expressed by the following formula: in represents the video features extracted in step 4, Represents a prototype feature.

6. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 6 includes more details: fusing the video features extracted in step 5 and the prototype features extracted in step 4 using a multi-layer perceptron to obtain video features with enhanced behavioral features, which can be expressed as follows: 。 7. The behavior-centric first-person perspective online action segmentation method according to claim 1, characterized in that: Step 7 includes the following details: For the video features extracted in step 6 with enhanced behavioral features, we not only hope to establish long-term dependency modeling, but also hope that the model will pay more attention to the intrinsic characteristics of the action behavior. This is to address the problem that the lack of subsequent context under online constraints limits the effectiveness of prior context for learning action semantics. Therefore, short-term behavior-related frames are selected for feature enhancement. This process is mainly presented in the training process. In the inference phase, when using the model for online action segmentation tasks, we use the video features with enhanced behavioral features extracted in step 6 and input them into the behavioral feature enhancement decoder based on the causal attention mechanism to obtain the final video features. We then input them into a fully connected layer to obtain the action classification corresponding to each frame and the action segmentation results. We then calculate the accuracy, edit distance, and F1 score respectively.

8. A behavior-centric first-person perspective online action segmentation system, characterized by: Specifically include: Acquisition unit: input the video that needs to be segmented online; I3D feature extractor: Use the pre-trained I3D feature extractor to extract frame-level video features for each frame in the video; Prototype feature learning transformer: Based on the trained prototype feature learning transformer based on the causal attention mechanism, the prototype features of each type of action are obtained; Behavior Feature Enhancement Encoder: The features extracted by the I3D feature extractor are input into the trained behavior feature enhancement encoder based on the causal attention mechanism, and the causal encoder module is used to extract video features without future information; Cosine similarity calculation unit: Calculates the cosine similarity between the video features without future information and the prototype features of each action category; obtains the prototype features that best match the video features based on the cosine similarity with the prototype, and obtains the common characteristics of behaviors in different environments; Multi-layer perceptron: Use a multi-layer perceptron to fuse the matched prototype features and video features to obtain video features with enhanced behavioral features; Behavior Feature Enhancement Decoder: The obtained video features with enhanced behavior features are input into the behavior feature enhancement decoder based on the causal attention mechanism to obtain the final video features, which are then input into a fully connected layer to obtain the first-person perspective online action classification results.

Images