High dynamic scene-oriented first view behavior simulation error detection method

Abstract

The invention discloses a first visual angle behavior simulation error detection method for a high dynamic scene, and belongs to the field of video quality evaluation. The method is further improved on the basis of a PDVC method in the field of dense video description proposed by ICCV2021. In order to solve the problem, on the basis of retaining a PDVC trunk detection normal form, an adaptive sampling module is introduced to screen key segments and suppress redundancy, scene adaptive visual angle embedding is proposed to explicitly model visual angle conditions and relieve domain deviation, and bidirectional cross-visual angle attention fusion is designed to realize mutual detection alignment and complementary evidence aggregation, so that the detection accuracy is improved. Therefore, the positioning and distinguishing capability of step-level error segments is improved.

Description

Technical Field

[0001] This invention belongs to the field of first-person perspective video understanding and cross-perspective imitation quality assessment. Specifically, it uses first-person perspective imitation videos and third-person perspective demonstration videos as input to perform step-level semantic alignment, time-segment localization, and error determination in the imitation process. Addressing the inherent problems of asynchronous acquisition, inconsistent rhythm, interference from redundant frames in long videos, and significant first / third-person perspective bias in demonstration-imitation videos, this invention proposes a cross-perspective adaptive sampling and scene-adaptive perspective alignment mechanism based on a pre-trained video representation and end-to-end temporal detection framework. Specifically, this invention introduces an adaptive sampling module to filter key segments from two long temporal sequences to suppress redundancy; proposes a scene-adaptive perspective embedding module to generate scene-varying perspective condition representations through a learnable perspective dictionary to reduce cross-perspective differences; and designs a bidirectional cross-perspective attention fusion module to achieve mutual alignment and complementary evidence aggregation of demonstration and imitation features. This outputs candidate step intervals and their "correct / incorrect" judgment results on the first-person perspective timeline, applicable to embodied intelligence scenarios such as industrial assembly training, medical operation quality control, human-machine collaboration, and service robots. Background Technology

[0002] With the development of wearable devices and humanoid robot technology, first-person perspective visual understanding has gradually become a research hotspot. Existing research has made significant progress in first-person perspective action classification, temporal action localization, and procedural activity recognition, but most of them only use video from a single perspective as input, assuming that the demonstration and execution come from the same camera perspective, which makes it difficult to deal with cross-view settings of "first-person imitation after third-person perspective demonstration" in real-world scenarios.

[0003] In scenarios such as industrial assembly training, nurses imitating standard medical procedures, or robots following human instruction, a common practice is to use a fixed third-view camera to film a demonstration video of standard operations, which is then imitated by trainees or robots wearing a first-view camera. In this case, it is necessary to judge the correctness of each step in the first-view execution process based on the third-view demonstration video. However, most existing error detection methods work in a single perspective: either they only utilize the first-view stream to learn online error detection, or they only compare the current execution with a standard template from a third-view perspective. Existing methods struggle to directly model cross-perspective demonstration-imitation pairs.

[0004] In this cross-view imitation scenario, at least three key challenges exist: 1. Asynchronous timing and inconsistent rhythm: Demonstration and imitation videos are typically captured asynchronously with varying durations, resulting in significant differences in execution rhythm. Simply aligning the timeline can lead to numerous mismatches, but "inconsistent duration" itself does not necessarily indicate an error. 2. Severe redundant frames: Long videos contain numerous pauses, preparation segments, or background footage. These redundant frames do not affect step determination but dilute attention, causing the detection model to be insensitive to a small number of keyframes. 3. Significant viewpoint bias: First-person perspective videos emphasize close-up hand-object interaction, while third-person perspective videos highlight full-body posture and global scene layout. The two differ systematically in appearance, motion, and imaging geometry. Directly splicing or simply fusing these two features can easily misjudge viewpoint differences as action differences.

[0005] Existing methods for temporal action detection and dense video description typically assume that the input is a single-stream video and perform temporal modeling and segment prediction using Transformer or deformable Transformer structures. Some works have also attempted to align simultaneous, synchronous multi-view videos, but most of them rely on strict synchronization or geometric calibration, which is difficult to apply to the "asynchronous, duration-inconsistent first-view–third-view demonstration / imitation pairs" that this invention focuses on.

[0006] Existing PDVC employs a deformable Transformer encoder-decoder and query-driven ensemble prediction paradigm to jointly predict multiple event segments (and their descriptions) in long videos. In this task, frozen first / third-view video features can be simply concatenated and input into the PDVC framework, with an additional error discrimination head inserted to output step-level error intervals. However, since demonstration-imitation videos are typically acquired asynchronously and have inconsistent pacing, long videos have a high proportion of redundant segments. Furthermore, there is a significant domain deviation between the first and third views. The aforementioned direct transfer method is susceptible to redundancy interference and misjudgment due to viewpoint differences, resulting in limited performance in locating and recognizing fine-grained error segments. Therefore, it is necessary to design a novel cross-view imitation error detection method that addresses temporal asynchrony, severe redundancy, and viewpoint domain deviation, capable of providing step intervals and their "correct / incorrect" judgments on the first-view timeline. Summary of the Invention

[0007] The purpose of this invention is to solve the key technical problems in cross-view third-person demonstration – first-person imitation error detection, and to introduce adaptive sampling, scene adaptive view embedding and bidirectional cross-view fusion mechanism while retaining the PDVC query set prediction and time segment generation paradigm, thereby realizing step-level interval positioning and “correct / incorrect” determination on the first-person time axis.

[0008] To achieve the above objectives, this invention proposes a first-person perspective behavior imitation error detection method for high dynamic scenes. The method includes, in sequence: a pre-trained video feature extraction module, an adaptive sampling module, a scene adaptive perspective embedding module, a bidirectional cross-view fusion module, and a step detection and error judgment module based on a deformable Transformer.

[0009] Step 1: The specific method for the pre-trained video feature extraction module is as follows:

[0010] S1.1 Acquire third-person perspective demonstration video sequence and first-person perspective imitation video sequence , , These represent the frame images or segments at the t-th time position from the two perspectives, respectively; , These represent the number of frames or segments in the demonstration and imitation sequences, respectively.

[0011] S1.2 uses a pre-trained video coding network to respectively... and Encode to obtain frame-by-frame or segment-by-segment feature sequences. , , where d is the feature dimension;

[0012] Step 2: Select key segments from the long sequence using an adaptive sampling module:

[0013] S2.1 Third-Perspective Feature Sequence By applying a self-attention layer and a feedforward network, a saliency score vector is obtained for each time position. ;

[0014] S2.2 uses the differentiable Gumbel Top-K sampling operator to process the saliency score vector; first, it generates a hard sampling index set. That is, selection The highest-scoring frame index is selected while preserving the normalized soft-selection distribution. ;

[0015] S2.3 for complete sequences Introducing residual gating:

[0016] According to the soft selection distribution Construct a weight vector with an approximate mean of 1. The calculation formula is as follows:

[0017] ;

[0018] in, This indicates the operation of scaling the soft score to a mean of 1. The hyperparameters for controlling the gating strength are then used; subsequently, the original feature sequence is... Perform position-by-position weighting and in the hard sampling index Feature collection is performed at the location to obtain a sparse demonstration feature sequence. :

[0019] ;

[0020] This step, while preserving keyframe features, allows gradient backpropagation to the scoring network via residual paths, ensuring training stability. Represents element-wise product;

[0021] S2.4, for first-person perspective emulation streams, takes into account that keyframes depend not only on the emulation itself but also on the comparison with the demonstration, and will... As a query, As key / value pairs, a cross-attention mechanism is used to compute the imitation-side saliency score. Similarly, Gumbel Top-K and residual gating are used to generate sparse feature sequences that mimic the side. ;

[0022] Step 3: Employ the scene adaptive perspective embedding module to explicitly model the differences between perspective conditions and scene conditions;

[0023] Step 4: Use the bidirectional cross-view fusion module to perform mutual inspection and alignment demonstrations and simulate evidence;

[0024] Step 5: The specific method of the step detection and error judgment module based on deformable Transformer is as follows: S5.1 fusing features Input a multi-layer deformable Transformer encoder to perform multi-scale aggregation on long time series to obtain multi-scale temporal features; S5.2 Set a fixed number of N learnable queries at the decoder end. Each query interacts with the encoded features through several decoding layers, gradually regressing to a candidate step interval on the first-person perspective timeline and its imitation correctness prediction; S5.3 Each query output includes: first-person perspective normalized start and end times. and tags Where 1 indicates an error and 0 indicates a correct imitation;

[0025] Step 6: Training Objectives and Optimization: To achieve end-to-end training, an ensemble prediction framework is adopted, establishing a one-to-one correspondence between the prediction set and the ground truth labeled step set through Hungarian matching. The overall loss is a weighted sum of the following components: 1) Step localization and description loss: including boundary regression loss, quality metric, and optional auxiliary description loss; 2) Imitation correctness classification loss: applying binary cross-entropy or focus loss to the "correct / incorrect" label of each candidate step; 3) Adaptive sampling regularization loss: including selection entropy loss and feature redundancy suppression loss; 4) Viewpoint embedding regularization loss: including attention entropy-uniformity loss and dictionary diversity loss; Step 7: Imitation Error Judgment and Result Output: The final model outputs a set of predictions. ,in: The normalized start and end times of this step on the first-view time axis obtained from the regression; Assign a confidence score to this step; For binary classification labels, if the output probability of the above error detection head is greater than a preset threshold, the step is judged as "error"; otherwise, it is judged as "correct".

[0026] Furthermore, in step 2.4, to avoid collapse to a few locations and suppress feature redundancy between selected segments, this module also introduces: 1) a selection entropy regularization term, which imposes a high entropy constraint on the Top-K selection distribution, guiding the model to cover richer time locations; 2) a redundancy suppression term based on variance and covariance, which implements variance lower bound and decorrelation constraints on the feature dimensions of the selected segments to prevent features from collapsing to low-dimensional subspaces; the specific method is as follows:

[0027] Select entropy regularization term For soft-selection distributions Applying high-entropy constraints encourages the model to explore more diverse temporal locations, avoiding focus on a very small number of frames; the calculation formula is:

[0028] ;

[0029] in, This represents the soft-choice distribution at time position t. Let be the numerical stability constant;

[0030] Redundancy suppression terms based on variance and covariance For the selected feature sequence Apply variance lower bound constraints and decorrelation constraints; the specific formulas are as follows:

[0031] ;

[0032] , The corresponding weights are represented by , and by jointly optimizing the above constraints, the generated sparse feature sequence has high information density while maintaining low redundancy.

[0033] Furthermore, the specific method of statement 3 is as follows:

[0034] S3.1 Set up a view dictionary Where M is the number of prototypes, and each row represents a common viewpoint – scene sub-factor; S3.2 sparse feature sequence After scaling, the query is used, where u represents ego or exo, and dictionary D serves as the key and value. Multi-head attention is used to calculate attention weights and outputs, resulting in a viewpoint embedding sequence aligned with the time position. S3.3 Encodes the viewpoint embedding sequence and time position. These are added together to the corresponding sparse feature sequence to obtain enhanced features with time-viewpoint conditions: View embedding is injected once before cross-view fusion and repeatedly injected at multiple levels of the subsequent multi-scale encoder to achieve multi-level modulation; S3.4 applies a divergence regularization to the attention distribution of the queried dictionary and a uniform distribution, and applies a diversity regularization to the dictionary row vectors after L2 normalization.

[0035] Furthermore, the specific method for step 4 is as follows:

[0036] S4.1 First-person perspective features For query, third-person perspective features Using key / value pairs, compute the evidence of "boundary and temporal structure" retrieved from the demo stream. S4.2 Features from a third-person perspective For query, first-person perspective features Using key / value pairs, we compute the evidence of "hand-object details and local causality" retrieved from the imitation stream. S4.3 introduces learnable gating on both branches:

[0037] , ;

[0038] in and The fusion is achieved by combining the concatenated local flow features and cross-flow evidence through a linear layer and a sigmoid function, with values ​​ranging from (0,1), thus enabling position-wise adaptive fusion ratios. S4.4 then averages the features from the two branches to obtain the fused cross-flow representation. This serves as the input for subsequent detection modules.

[0039] The present invention has the following beneficial effects:

[0040] 1. A unified modeling approach for cross-view error detection tasks combining "third-person demonstration" and "first-person imitation" can simultaneously provide step intervals and their "correct / incorrect" labels on the first-person timeline, making it suitable for automated evaluation needs in scenarios such as industrial training and medical quality control.

[0041] 2. The adaptive sampling module combines Top-K hard selection and residual gating with selection entropy and redundancy suppression regularization to significantly compress redundant frames while maintaining end-to-end trainability, and strengthens the focus on key action segments, thereby improving the robustness of cross-view alignment and error detection.

[0042] 3. The scene-adaptive view embedding module generates view embeddings that change with the scene through dictionary-based attention and injects features at multiple levels to explicitly model the domain differences between the first and third views. This makes the similarity distribution between the first two views more concentrated and smoother, alleviating the problem of "mistaking view differences for operational differences".

[0043] 4. The bidirectional cross-view fusion module uses symmetrical cross-attention and learnable gating to maintain the stability of the current viewpoint while introducing complementary evidence from the opposite viewpoint. It increases cross-view coupling near the action boundary and suppresses unnecessary information exchange in the background area, thus balancing alignment capability and viewpoint robustness.

[0044] 5. Experiments on the asynchronous first-person to third-person perspective simulation dataset EgoMe show that, compared with existing methods that only use simple splicing or unidirectional conditions, the present invention achieves significant improvements in both AUPRC and average tIoU metrics for error segment detection.

[0045] Attached Description

[0046] Figure 1 This describes the specific network architecture for this method.

[0047] Figure 2 This demonstrates the effectiveness of this method in several example action videos; among them, Figure 2 (a) is a visualization of the frame saliency of the adaptive sampling module of the present invention on different video samples, with the dark area representing the selected key segments; Figure 2 (b) is a schematic diagram of the step-level detection results of the method of the present invention and the comparative method on a certain imitation task, wherein the green bars represent correct steps and the red bars represent detected incorrect steps. Detailed Implementation

[0048] Taking a third-person perspective demonstration video and a first-person perspective imitation video as input, the invention outputs multiple process step intervals and their error judgment labels on the first-person perspective timeline. The invention includes the following modules:

[0049] The method proposed in this invention is generally divided into three stages: the first stage is the cross-view basic representation and adaptive sampling stage; the second stage is the scene adaptive view alignment and bidirectional cross-view fusion stage; the third stage is the query-based step detection and joint optimization stage; for ease of understanding, the main symbols and abbreviations involved in the figure and the following text are explained: (1) Input and length: This indicates a third-person perspective demonstration video sequence. This indicates a first-person perspective imitating a video sequence; , These represent the frame images (or segments) at the t-th time position from the two perspectives, respectively. , These represent the duration (number of frames or number of segments) of the demonstration and imitation sequences, respectively, and the two are allowed to be unequal. (2) Features and dimensions: , These represent the third-view / first-view temporal feature sequences extracted by the pre-trained video coding network, respectively. The feature dimension representing a single time location. (3) Adaptive sampling: , These represent the saliency scores calculated for each time position in the third-person / first-person perspective (used to measure the importance of that position to the step determination). , These represent the number of key segments selected from the demonstration sequence and the imitation sequence, respectively. , These represent the sparse key feature sequences of the third-view / first-view perspectives obtained after Top-K selection and residual gating, respectively. (4) Scene-adaptive view embedding: A dictionary of learnable viewpoints (composed of multiple viewpoint / scene prototype vectors); Indicates the number of dictionary prototypes; , These represent the third-person / first-person perspective conditional embedding sequences generated through the "key feature query dictionary"; , This indicates the time position code corresponding to the viewpoint; , This represents the enhanced feature sequence after injecting time location encoding and viewpoint embedding. (5) Bidirectional cross-view fusion: This indicates cross-view evidence features obtained by "first-person perspective querying third-person perspective" (emphasizing structural clues such as step sequence / boundaries); This indicates cross-view evidence features obtained by querying a first-view perspective from a third-person perspective (with an emphasis on hand-object details / local causal clues). , This represents the learnable gating coefficient (ranging from 0 to 1, used to control the fusion ratio of features from this stream to cross-view evidence). , This indicates the two-way characteristics after gating fusion; This represents the final cross-view fusion representation obtained through aggregation. (6) Step detection and output (PDVC backbone paradigm): This represents the multi-scale fused features obtained after passing through the basic encoder; , These represent multi-scale temporal location embedding and viewpoint embedding, respectively. This represents the number of queries that the decoding end can learn, i.e., the maximum number of candidate steps that can be output; for the th A query that outputs the start and end positions of the step intervals on the first-person timeline. (Normalized time or frame / timestamp representation can be used) and imitation correctness determination (e.g., binary classification: correct / incorrect). (7) Training constraints: The training objective of this invention is composed of the dense subtitle loss in PDVC, the correct / incorrect classification error, and the sampling stability and redundancy removal constraints, as well as the viewpoint dictionary usage equalization and prototype diversity constraints, etc.; among which the sampling constraints are used to avoid the collapse of key segment selection and improve diversity, and the viewpoint dictionary constraints are used to avoid dictionary degradation and stabilize the cross-viewpoint modulation effect.

[0050] The prior art problem solved by this invention and the corresponding technical solution are as follows:

[0051] Firstly, addressing the issues of asynchronous acquisition, inconsistent rhythm, and high proportion of redundant segments in long videos between demonstration and imitation videos, existing detection methods that directly use uniform sampling or full input are easily affected by numerous pauses, preparation actions, and background frames, leading to diluted evidence of key steps and unstable cross-view alignment. To solve this problem, this invention designs an adaptive sampling module: calculating saliency scores for temporal features from both third-person and first-person perspectives, and employing a differentiable Top-K selection strategy to select several key segments from the long sequence; simultaneously, a residual gating mechanism is introduced, using soft selection weights to weighted retain of features across the entire sequence, enabling the model to obtain sparse key segments while maintaining continuous transmission of training gradients; furthermore, to prevent the sampling distribution from collapsing to a few positions and suppress redundancy between selected segments, this invention applies entropy regularization constraints to the selection distribution and redundancy removal constraints to the selected features, thereby improving key segment coverage and representation diversity, and enhancing the stability of cross-view alignment and error detection.

[0052] Secondly, regarding the significant differences in appearance and motion patterns between first-person and third-person perspectives, existing methods that simply stitch together features from both perspectives or use fixed perspective identifier vectors are insufficient to characterize the "systematic perspective bias that changes with the scene," easily misjudging perspective differences as action differences and leading to false detections. To address this issue, this invention proposes a scene-adaptive perspective embedding module: constructing a learnable perspective dictionary (composed of multiple perspective prototype vectors), and using the current temporal features as a query to interact with this dictionary through attention, generating perspective conditional embeddings that change with the scene and time; the perspective embeddings and temporal position encodings are jointly injected into the features of both perspectives, and can be repeatedly injected at multiple levels of the encoder to achieve continuous modulation; simultaneously, to ensure the coverage and separability of the dictionary prototypes, this invention applies uniformity / high entropy constraints to the dictionary attention distribution and imposes constraints on the diversity among dictionary prototypes, thereby reducing cross-perspective domain bias in the feature space and improving the reliability of cross-view comparisons.

[0053] Thirdly, regarding the difficulty of cross-view information fusion, existing one-way conditional fusion often suffers from information bias and insufficient evidence: the third-person perspective is better at providing step sequence and action boundary clues, while the first-person perspective is better at providing hand-object contact details and local causal evidence. If only one-way transmission is used, it is easy to cause inaccurate boundaries or insufficient erroneous evidence. To solve this problem, this invention designs a bidirectional cross-view fusion module: at the key segment level, bidirectional cross-attention interaction of "first-person perspective querying third-person perspective" and "third-person perspective querying first-person perspective" is executed respectively to achieve mutual retrieval and mutual constraint; and a learnable gating fusion strategy is introduced to adaptively generate fusion coefficients based on the features of the current perspective and cross-view evidence, dynamically adjusting the information exchange intensity for different time positions / channels, thereby enhancing cross-view coupling in action boundaries and key interaction areas, suppressing invalid exchanges in background and non-critical areas, and thus improving the accuracy of step boundary positioning and the ability to aggregate erroneous evidence.

[0054] Step 1: Acquire Demo – Imitate Video Pairs and Extract Basic Features: S1.1 Acquire Third-Perspective Demo Video Sequences and first-person perspective imitation video sequence The start and end times of the two sequences may differ from the total duration. S1.2 A pre-trained video coding network (such as a pre-trained temporally sensitive converter or a 3D CNN–Transformer hybrid model) is used to respectively... and Encode to obtain frame-by-frame or segment-by-segment feature sequences. , , where d is the feature dimension. In this invention, the encoding network maintains frozen parameters to reduce training costs and leverage its generalization ability on temporal localization tasks.

[0055] Step 2: Adaptive sampling module selects key segments from long sequences: To suppress redundant frames in demonstration and imitation videos and increase the density of key action segments, this invention designs adaptive sampling for the two feature sequences separately. S2.1 For third-view feature sequences By applying a self-attention layer and a feedforward network, a saliency score vector is obtained for each time position. S2.2 Utilizes the differentiable Gumbel Top-K sampling operator, based on... choose The time indices with the highest saliency constitute the hard-sampling index set; simultaneously, the normalized soft-selection distribution is retained as a reference for subsequent residual gating. S2.3 To alleviate the gradient sparsity problem caused by relying solely on hard sampling, this invention... Introducing residual gating: Based on the soft-selection distribution, a weight vector with an approximate mean of 1 is constructed. The original features are then scaled position-by-position, and features are collected at the selected Top-K positions to obtain a sparse demonstration feature sequence. S2.4 For first-person perspective emulation streams, considering that keyframes depend not only on the emulation itself but also on the comparison with the demonstration, this invention will... As a query, As key / value pairs, a cross-attention mechanism is used to compute the imitation-side saliency score. Similarly, Gumbel Top-K and residual gating are used to generate sparse feature sequences that mimic the side. To avoid collapse to a few locations and suppress feature redundancy among selected segments, this module also introduces: 1) a selection entropy regularization term, which imposes a high entropy constraint on the Top-K selection distribution to guide the model to cover a wider range of time locations; 2) a redundancy suppression term based on variance and covariance, which imposes a lower bound on the variance and decorrelation constraint on the feature dimension of the selected segments to prevent features from collapsing to a low-dimensional subspace.

[0056] Step 3: Scene-Adaptive Viewpoint Embedding Module, Explicitly Modeling Viewpoint Conditions and Scene Differences: To alleviate the feature domain bias between the first-person and third-person perspectives, this invention constructs a scene-adaptive viewpoint dictionary and generates viewpoint embeddings that change with the scene through an attention mechanism. S3.1 Setting the Viewpoint Dictionary Where M is the number of prototypes, and each row represents a common viewpoint-scene sub-factor, such as "close-up hand-object interaction" or "full-body motion structure". S3.2 sparse feature sequences (u takes ego or exo) After scaling, it is used as the query, and dictionary D is used as the key and value. Multi-head attention is used to calculate the attention weights and output, resulting in a viewpoint embedding sequence aligned with the time position. S3.3 will be encoded with time and location. These are added to the corresponding features to obtain enhanced features with time-view conditions: Viewpoint embedding is injected once before cross-view fusion and repeatedly injected at multiple levels of the subsequent multi-scale encoder to achieve multi-level modulation. S3.4 To avoid the viewpoint dictionary degenerating into a few sharp prototypes, this invention applies a divergence regularization to the attention distribution of the queried dictionary, similar to a uniform distribution, to encourage higher entropy; simultaneously, after L2 normalization of the dictionary row vectors, a diversity regularization is applied to promote near-orthogonality between prototypes. Through these mechanisms, the model no longer relies solely on "unconditional" visual features for alignment, but explicitly knows whether the current feature comes from a first or third viewpoint, and in which viewpoint subspace it is located, thereby reducing the impact of viewpoint domain bias on false detections.

[0057] Step 4: Two-way cross-view fusion module, mutual inspection and alignment of demonstration and imitation evidence. This is used to obtain sparse feature sequences with time-view conditions. and Subsequently, this invention achieves semantic alignment and complementary evidence aggregation through bidirectional cross-view attention. S4.1 First-view features For query, third-person perspective features Using key / value pairs, compute the evidence of "boundary and temporal structure" retrieved from the demo stream. S4.2 Features from a third-person perspective For query, first-person perspective features Using key / value pairs, we compute the evidence of "hand-object details and local causality" retrieved from the imitation stream. S4.3 To prevent one-sided evidence from overwhelming the information in this stream, this invention introduces learnable gating on both branches: , ,in and The fusion is achieved by combining the concatenated local features and cross-flow evidence through a linear layer and a sigmoid function, with values ​​ranging from (0,1), thus enabling position-wise adaptive fusion scaling. S4.4 Averaging the features from the two branches yields the fused cross-view representation. This serves as input for subsequent detection modules. Through the aforementioned bidirectional fusion, the third-person perspective provides more reliable clues about the sequence of steps and action boundaries, while the first-person perspective supplements key objects, gesture details, and local contact information. The two perspectives mutually verify each other within the feature space, which helps to accurately locate erroneous steps and determine their nature.

[0058] Step 5: Step detection and imitation error judgment based on deformable Transformer: S5.1 Fusing features Inputting a multi-layer deformable Transformer encoder, the long time series is aggregated at multiple scales to obtain multi-scale temporal features. S5.2 A fixed number of N learnable queries are set at the decoder. Each query interacts with the encoded features through several decoding layers, gradually regressing to a candidate step interval on the first-person perspective timeline and its imitation correctness prediction. S5.3 Each query output includes: the first-person perspective normalized start and end times. and tags , where 1 indicates an error and 0 indicates a correct imitation.

[0059] Step 6: Training Objectives and Optimization: To achieve end-to-end training, this invention employs an ensemble prediction framework, establishing a one-to-one correspondence between the prediction set and the ground truth labeled step set through Hungarian matching. The overall loss is a weighted sum of the following components: 1) Step localization and description loss: including boundary regression loss and quality metrics (such as IoU correlation loss) as well as optional auxiliary description loss to improve temporal localization accuracy; 2) Mimicking correctness classification loss: applying binary cross-entropy or focus loss to the "correct / incorrect" label of each candidate step; 3) Adaptive sampling regularization loss: including selection entropy loss and feature redundancy suppression loss; 4) Viewpoint embedding regularization loss: including attention entropy-uniformity loss and dictionary diversity loss. By jointly optimizing the above losses, the model can simultaneously learn: how to select and align key segments in long-term asynchronous videos, how to eliminate viewpoint domain bias, and accurately find erroneous steps on the first-viewpoint timeline.

[0060] This invention innovatively proposes a first-person perspective imitation error detection method based on scene-adaptive viewpoint alignment and cross-view fusion, used to achieve step-level interval localization and error judgment in asynchronous, long-duration third-person perspective demonstration videos and first-person perspective imitation videos. Addressing three core challenges commonly found in cross-view imitation scenarios—dense redundant segments in long videos, significant domain deviation between first and third perspectives, and difficulty in cross-view information fusion—this invention systematically designs an adaptive sampling module, a scene-adaptive viewpoint embedding module, and a bidirectional cross-view fusion module from three perspectives: "key segment selection," "viewpoint difference modulation," and "cross-view evidence alignment." Through the synergistic effect of these modules, this invention achieves refined analysis of the cross-view imitation process, providing an accurate and reliable automated imitation error detection solution for scenarios such as embodied intelligence, industrial training, medical quality control, and human-machine collaboration.

Claims

1. A method for detecting errors in first-person perspective behavior imitation in highly dynamic scenes, the method comprising: The pre-trained video feature extraction module, adaptive sampling module, scene adaptive view embedding module, bidirectional cross-view fusion module, and step detection and error judgment module based on deformable Transformer; Step 1: The specific method for the pre-trained video feature extraction module is as follows: S1.1 Acquire third-person perspective demonstration video sequence and first-person perspective imitation video sequence , , These represent the frame images or segments at the t-th time position from the two perspectives, respectively; , These represent the number of frames or segments in the demonstration and imitation sequences, respectively. S1.2 uses a pre-trained video coding network to respectively... and Encode to obtain frame-by-frame or segment-by-segment feature sequences. , , where d is the feature dimension; Step 2: Select key segments from the long sequence using an adaptive sampling module: S2.1 Third-Perspective Feature Sequence By applying a self-attention layer and a feedforward network, a saliency score vector is obtained for each time position. ; S2.2 uses the differentiable Gumbel Top-K sampling operator to process the saliency score vector; first, it generates a hard sampling index set. That is, selection The highest-scoring frame index is selected while preserving the normalized soft-selection distribution. ; S2.3 for complete sequences Introducing residual gating: According to the soft selection distribution Construct a weight vector with an approximate mean of 1. The calculation formula is as follows: ； in, This indicates the operation of scaling the soft score to a mean of 1. The hyperparameters for controlling the gating strength are then used; subsequently, the original feature sequence is... Perform position-by-position weighting and in the hard sampling index Feature collection is performed at the location to obtain a sparse demonstration feature sequence. : ； This step, while preserving keyframe features, allows gradient backpropagation to the scoring network via residual paths, ensuring training stability. Represents element-wise product; S2.4, for first-person perspective emulation streams, takes into account that keyframes depend not only on the emulation itself but also on the comparison with the demonstration, and will... As a query, As key / value pairs, a cross-attention mechanism is used to compute the imitation-side saliency score. Similarly, Gumbel Top-K and residual gating are used to generate sparse feature sequences that mimic the side. ; Step 3: Employ the scene adaptive perspective embedding module to explicitly model the differences between perspective conditions and scene conditions; Step 4: Use the bidirectional cross-view fusion module to perform mutual inspection and alignment demonstrations and simulate evidence; Step 5: The specific method of the step detection and error judgment module based on deformable Transformer is as follows: S5.1 fusing features The input is a multi-layer deformable Transformer encoder, which performs multi-scale aggregation on long time series to obtain multi-scale temporal features; S5.2 sets a fixed number of N learnable queries at the decoder, each query interacts with the encoded features through several decoding layers, gradually regressing to a candidate step interval on the first-view timeline and its imitation correctness prediction; S5.3 each query output includes: first-view normalized start and end times. and tags Where 1 indicates an error and 0 indicates a correct imitation; Step 6: Training Objectives and Optimization: To achieve end-to-end training, an ensemble prediction framework is adopted, establishing a one-to-one correspondence between the prediction set and the ground truth labeled step set through Hungarian matching. The overall loss is a weighted sum of the following components: 1) Step localization and description loss: including boundary regression loss, quality metric, and optional auxiliary description loss; 2) Imitation correctness classification loss: applying binary cross-entropy or focus loss to the "correct / incorrect" label of each candidate step; 3) Adaptive sampling regularization loss: including selection entropy loss and feature redundancy suppression loss; 4) Viewpoint embedding regularization loss: including attention entropy-uniformity loss and dictionary diversity loss; Step 7: Imitation Error Judgment and Result Output: The final model outputs a set of predictions. ,in: The normalized start and end times of this step on the first-view time axis obtained from the regression; Assign a confidence score to this step; For binary classification labels, if the output probability of the above error detection head is greater than a preset threshold, the step is judged as "error"; otherwise, it is judged as "correct".

2. The method for detecting first-person perspective behavior imitation errors in highly dynamic scenes as described in claim 1, characterized in that, To avoid collapsing to a few locations and suppress feature redundancy between selected segments in step 2.4, this module also introduces: 1) a selection entropy regularization term, which imposes a high entropy constraint on the Top-K selection distribution, guiding the model to cover richer time locations; 2) a redundancy suppression term based on variance and covariance, which implements variance lower bound and decorrelation constraints on the feature dimensions of the selected segments to prevent features from collapsing to low-dimensional subspaces; the specific methods are as follows: Select entropy regularization term For soft-selection distributions Imposing high-entropy constraints encourages the model to explore more diverse temporal locations, avoiding focusing on a very small number of frames; The calculation formula is: ； in, This represents the soft-choice distribution at time position t. Let be the numerical stability constant; Redundancy suppression terms based on variance and covariance For the selected feature sequence Apply variance lower bound constraints and decorrelation constraints; the specific formulas are as follows: ； , The corresponding weights are represented by , and by jointly optimizing the above constraints, the generated sparse feature sequence has high information density while maintaining low redundancy.

3. The method for detecting first-person perspective behavior imitation errors in highly dynamic scenes as described in claim 1, characterized in that, The specific method of the third point is as follows: S3.1 Set up a view dictionary Where M is the number of prototypes, and each row represents a common viewpoint – scene sub-factor; S3.2 will sparse feature sequence After scaling, the query is used, where u represents ego or exo, and dictionary D serves as the key and value. Multi-head attention is used to calculate attention weights and outputs, resulting in a viewpoint embedding sequence aligned with the time position. ; S3.3 embeds the viewpoint sequence with time position encoding. These are added together to the corresponding sparse feature sequence to obtain enhanced features with time-viewpoint conditions: ; View embedding is injected once before cross-view fusion and repeatedly injected at multiple levels of the subsequent multi-scale encoder to achieve multi-level modulation; S3.4 applies a divergence regularization to the attention distribution of the queried dictionary and a uniform distribution, and applies a diversity regularization after L2 normalization of the dictionary row vectors.

4. The method for detecting first-person perspective behavior imitation errors in highly dynamic scenes as described in claim 1, characterized in that, The specific method for step 4 is as follows: S4.1 First-person perspective features For query, third-person perspective features Using key / value pairs, compute the "boundary and temporal structure" evidence retrieved from the demo stream. ; S4.2 Features from a Third Perspective For query, first-person perspective features Using key / value pairs, we compute the "hand-object detail and local causality" evidence retrieved from the imitation stream. ; S4.3 introduces learnable gating on both branches: ， ； in and The feature is obtained by splicing the local flow features and cross-flow evidence through a linear layer and a Sigmoid function, with a value range of (0,1), to achieve an adaptive fusion ratio for each position. S4.4 averages the features of the two branches to obtain the fused cross-view representation. This serves as the input for subsequent detection modules.

Images