A chemical experiment video online action segmentation method based on view angle robust learning
By employing a viewpoint-robust learning approach and designing viewpoint feature branches and action feature branches, the accuracy and generalization issues of online temporal action segmentation methods under viewpoint changes are addressed. Stable and accurate action segmentation is achieved in chemical experiment videos, improving the model's adaptability and cross-viewpoint recognition performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING NORMAL UNIVERSITY
- Filing Date
- 2025-10-31
- Publication Date
- 2026-07-24
AI Technical Summary
Existing online temporal action segmentation methods suffer from decreased segmentation accuracy and insufficient generalization ability when faced with changes in perspective, making it impossible to maintain stability and accuracy in real teaching scenarios.
We employ a viewpoint robust learning approach, using a dual-branch design of viewpoint feature branches and action feature branches to extract viewpoint-related contextual information and viewpoint-independent action information, respectively. We also introduce multi-level supervision strategies, such as viewpoint classification loss and cross-viewpoint consistency alignment loss, to enhance the model's adaptability to viewpoint changes.
This improves the robustness and cross-view generalization performance of online action segmentation in chemical experiment videos, achieving stable and accurate action segmentation and providing reliable technical support for experimental process monitoring and operation evaluation in intelligent education scenarios.
Smart Images
Figure CN121305445B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of video understanding and intelligent education technology, specifically to an online action segmentation method for chemical experiment videos based on viewpoint robust learning. Background Technology
[0002] Chemical experiment teaching is an indispensable and crucial component of the chemistry curriculum. Through hands-on activities, students can intuitively understand chemical principles, master experimental skills, and cultivate scientific inquiry abilities. With the continuous advancement of intelligent education and the informatization of experimental teaching, video-based chemical experiment process analysis technology is gradually becoming an important tool for teaching evaluation, experimental guidance, and process monitoring. A chemical experiment typically consists of a series of continuous and logically related operations, such as adding liquids, heating, and stirring. Each operation not only has a clear purpose but also often depends on the sequence and temporal relationship of preceding and following steps. Therefore, understanding the experimental process requires full consideration of the logical connections and contextual dependencies between operations. Online Temporal Action Segmentation (ATS) technology can dynamically predict the action category of each frame based on historical experimental video information, enabling real-time analysis of the experimental process. This not only supports accurate understanding of the experimental process but also provides a decision-making basis for subsequent intelligent interaction, operational guidance, and experimental evaluation.
[0003] Current online temporal action segmentation methods are typically based on modules such as causal temporal convolution, causal attention mechanisms, or long short-term memory networks. Their feature fusion relies solely on information from historical frames to ensure the causality and real-time performance of the model. These methods generally rely on a fixed viewpoint assumption, meaning that the videos used in the training and testing phases are taken from the same shooting angle or scene configuration. Therefore, the research focus is mainly on the temporal modeling of actions. However, in real-world teaching scenarios, due to differences in equipment layout, lab bench locations, and camera positions in different school laboratories, chemical experiments are often conducted under different environments and angles, resulting in significant differences in the visual representation of the same action. For existing online temporal action segmentation methods, this viewpoint variation weakens the consistency of the model in action representation learning, making the recognition results highly sensitive to the shooting angle. This leads to problems such as decreased segmentation accuracy, blurred action boundaries, and limited cross-scene generalization ability. This vulnerability to viewpoint variation not only affects the stability of the model in complex environments but also limits its application value in real-world scenarios.
[0004] A search revealed Chinese invention patent CN120451853A, which discloses an online action segmentation method based on uncertainty in the field of video action analysis technology. This invention employs a strategy of dynamically changing the network according to uncertainty, achieving efficient and accurate online action segmentation even when future information is missing. The invention utilizes a contextualization method based on the principle of uncertainty. This method comprises two key components: first, a context aggregation transformer, which combines a joint autoencoder and an autoregressive design to enhance the contextualization of frame-level features, effectively overcoming the limitations caused by the lack of subsequent context in streaming video; second, evidence-based action contextualization, which uses evidence-based deep learning to quantify the uncertainty in model decisions while correcting ambiguous predictions based on previous context. This approach supplements contextual information and effectively overcomes the problems existing in the prior art. This invention can be widely applied to scenarios involving first-person perspective video action analysis, such as human behavior monitoring based on wearable devices, and understanding human actions in augmented reality and virtual reality scenarios.
[0005] The technical comparison between this application and the aforementioned patent is as follows:
[0006] The patent "An Uncertainty-Based First-Perspective Online Action Segmentation Method" focuses on the uncertainty of prediction caused by the lack of future context (online settings), improving the accuracy of online segmentation through uncertainty quantification and evidence-based smoothing and correction mechanisms. Its core lies in "how to adaptively change the network output using uncertainty when future information is missing." This invention primarily addresses the impact of perspective changes on the generalization and accurate recognition of action segmentation models, especially considering the free-viewpoint characteristics common in teaching scenarios such as chemical experiments. It proposes a learning mechanism for perspective-independent action representations and a cross-perspective consistency mechanism, thereby reducing dependence on fixed perspectives. The two methods differ significantly in target localization.
[0007] II. The patent "An Online Action Segmentation Method Based on Uncertainty from a First-Perspective" utilizes deep learning to estimate prediction uncertainty and smooths or corrects predictions based on this uncertainty. This invention employs a dynamic correction strategy based on short-term historical segmentation results. It inputs viewpoint-independent action representations into the online segmentation network to generate probabilities, then compares and smooths these probabilities with short-term historical classification results, suppressing short-term noise and over-segmentation. The two methods differ significantly in their approaches to enhancing temporal stability.
[0008] Third, the patent "A First-Perspective Online Action Segmentation Method Based on Uncertainty" proposes modules such as a context aggregation transformer, a causal encoder / predictive decoder, and evidence-based uncertainty computation, focusing on how to compensate for the lack of future information in online scenarios by predicting future features. In contrast, this invention systematically proposes a series of modules and losses for "viewpoint invariance" learning (viewpoint classification loss, viewpoint contrast loss, reconstruction loss, action prototype classification loss, cross-viewpoint attention consistency loss, etc.), and architecturally distinguishes between viewpoint branches and action branches, introducing action prototype and cross-viewpoint alignment training constraints, focusing on obtaining viewpoint-independent action representations during the training phase. The two differ significantly in design details and training objectives. Summary of the Invention
[0009] This invention provides an online action segmentation method for chemical experiment videos based on viewpoint robust learning, aiming to solve the problems of decreased segmentation accuracy and insufficient generalization ability caused by viewpoint changes in real teaching scenarios in existing technologies. Compared with existing methods, this invention explicitly introduces a viewpoint-independent action representation learning mechanism, enabling the model to maintain a consistent understanding of the same action under diverse viewpoints, providing more reliable technical support for experimental process monitoring, operation evaluation, and auxiliary teaching in intelligent education scenarios.
[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0011] An online action segmentation method for chemical experiment videos based on viewpoint robust learning, characterized by the following steps:
[0012] S1. Perform frame-level sampling on the real-time chemical experiment video stream and obtain the spatiotemporal feature sequence of consecutive frames through a pre-trained visual feature extraction network.
[0013] S2. Perform feature separation on the spatiotemporal feature sequence to obtain view-dependent context representation features and view-independent action representation features;
[0014] S3. Input the viewpoint-independent action representation into the online temporal segmentation network, and dynamically adjust the classification probability by combining the short-term historical segmentation results to obtain the final temporal segmentation result of the current frame.
[0015] As a preferred technical solution of the present invention: In step S1, the step of extracting the spatiotemporal feature sequence from the real-time chemical experiment video stream specifically includes:
[0016] S11. Perform frame-level sampling on the input video stream;
[0017] S12. Several consecutive frames after sampling are combined into a time segment. The time segment is then encoded using a pre-trained spatiotemporal feature extraction network to obtain spatiotemporal features that incorporate short-term spatiotemporal changes.
[0018] S13. The above-mentioned spatiotemporal features are retained and recorded in the last frame of the time sequence segment to ensure that the feature generation process satisfies causal constraints and forms a spatiotemporal feature sequence in the continuous video stream.
[0019] As a preferred technical solution of the present invention: in step S12, sampling is performed by uniform sampling when performing frame-level sampling.
[0020] As a preferred technical solution of the present invention: In step S2, the step of learning view-related context representations and view-independent action representations from the spatiotemporal feature sequence specifically includes:
[0021] S21. Input the original spatiotemporal feature sequence into the viewpoint feature learning branch and the action feature learning branch respectively, so as to learn the viewpoint-related context representation and the viewpoint-independent action representation at the same time.
[0022] S22. In the viewpoint feature learning branch, the viewpoint feature encoder is used to compress the input spatiotemporal feature sequence, extract the viewpoint-related context representation, and then restore it to the original feature dimension through the viewpoint feature decoder.
[0023] S23. In the action feature learning branch, the action feature encoder is used to compress the input spatiotemporal feature sequence, extract the viewpoint-independent action representation, and restore it to the original feature dimension through the action feature decoder.
[0024] S24. Sum the outputs of the two branches element by element to obtain the reconstructed spatiotemporal feature sequence. Apply a reconstruction loss function between the original spatiotemporal feature sequence and the reconstructed spatiotemporal feature sequence to ensure that key spatiotemporal semantic information is preserved during feature separation and compression.
[0025] As a preferred technical solution of the present invention: In step S22, the step of extracting view-related contextual representations in the view feature learning branch specifically includes:
[0026] S221. Perform adaptive temporal feature aggregation on the input spatiotemporal feature sequence, dynamically adjust the aggregation weights according to the context information of historical frames, and generate the viewpoint feature representation of the current frame.
[0027] S222. Input the view feature representation generated in step S221 into the view perception embedding mapping module to obtain the view classification probability distribution of the current frame. Calculate the view classification loss based on the real view label to constrain the model to learn discriminative view features.
[0028] S223. Input the viewpoint feature representations generated from different videos into a contrastive viewpoint discriminator, and calculate the viewpoint contrast loss by constructing positive and negative sample pairs.
[0029] S224. Jointly optimize the viewpoint classification loss and viewpoint contrast loss to guide the viewpoint feature learning branch to fully capture viewpoint-related contextual information in the video.
[0030] As a preferred technical solution of the present invention: In step S23, the step of extracting viewpoint-independent action representations in the action feature learning branch specifically includes:
[0031] S231. Establish action prototype representations to learn fragment-level action features of different operations;
[0032] S232. Input the action prototype representation into the self-attention mechanism module to capture the intrinsic correlation between different action prototypes;
[0033] S233. The viewpoint-independent action representation is fused with the action prototype representation updated by the self-attention mechanism module through cross-attention operation and output as a viewpoint-independent action representation to enhance the semantic consistency of frame-level features within different actions.
[0034] S234. Input the viewpoint-independent action representation output in step S233 into the self-attention module to establish frame-level temporal correlation, and apply frame-level classification loss to the output viewpoint-independent action representation.
[0035] S235. Perform cross-attention operation again between the action prototype representation output in step S232 and the view-independent action representation output in step S234 to further enhance the semantic alignment effect between the action prototype and the view-independent action representation, while retaining the attention score.
[0036] S236. Apply cross-view consistency alignment loss to the attention scores of the same experimental video under different viewpoints to enhance the cross-view robustness of the action prototype.
[0037] S237. Apply action-level classification loss to the action prototypes output by the cross-attention mechanism to improve the cross-view discriminativeness of the action prototypes.
[0038] As a preferred technical solution of the present invention: the view feature decoder and the action feature decoder are view feature decoders and action feature decoders constructed by causal temporal convolution, respectively.
[0039] As a preferred technical solution of the present invention: In step S3, the step of obtaining the final segmentation result based on the viewpoint-independent action representation features specifically includes:
[0040] S31. Input the viewpoint-independent action representation into the online temporal segmentation network to generate the action classification probability of the current frame;
[0041] S32. Compare the action classification probability of the current frame generated in S31 with the classification results of the short-term history, and suppress short-term noise fluctuations through a dynamic smoothing mechanism based on the short-term history.
[0042] S33. The classification probability, after being corrected by a dynamic smoothing mechanism based on short-term history, is selected with the highest probability to obtain the final action segmentation result of the current frame.
[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0044] 1. A dual-branch design of view feature branch and action feature branch is adopted, in which the view feature branch captures view-related contextual information, and the action feature branch extracts view-independent action information, thereby achieving effective separation of action semantic information.
[0045] 2. A viewpoint-independent action representation learning mechanism is introduced to enhance the model's adaptability to viewpoint changes, thereby improving the robustness and cross-viewpoint generalization performance of online action segmentation in chemical experiment videos.
[0046] 3. Design a multi-level supervision strategy, including viewpoint classification loss, cross-viewpoint consistency alignment loss, and action-level classification loss, to constrain model learning from multiple perspectives and ensure the discriminativeness and effectiveness of viewpoint-independent action features.
[0047] 4. It can achieve stable and accurate online motion segmentation for chemical experiment videos from different perspectives, providing reliable data support for the analysis of chemical experiment processes, operation evaluation, and intelligent teaching assistance in real-world scenarios. Attached Figure Description
[0048] Figure 1 This is a model framework diagram for an online action segmentation method for chemical experiment videos based on viewpoint robust learning.
[0049] Figure 2 This is a flowchart illustrating how spatiotemporal feature sequences are decomposed into viewpoint-dependent contextual representation features and viewpoint-independent action representation features in this invention.
[0050] Figure 3 This is a flowchart of the perspective representation learning module for learning perspective-related context representations in this invention.
[0051] Figure 4 This is a flowchart of the action representation learning module for learning viewpoint-independent action representations in this invention;
[0052] Figure 5This is a comparison of the performance indicators of the present invention and existing technologies in chemical experiment videos from different perspectives. Detailed Implementation
[0053] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0054] Reference Figure 1 This invention proposes an online action segmentation method for chemical experiment videos based on viewpoint robust learning, comprising the following steps:
[0055] Step S1: Perform frame-level sampling on the real-time chemical experiment video stream and obtain the spatiotemporal feature sequence of consecutive frames through a pre-trained visual feature extraction network.
[0056] For the input real-time chemical experiment video stream, uniform sampling is first performed during real-time reception to reduce the computational load on the model and improve processing speed.
[0057] For example, in a 25 FPS video stream, 5 frames are sampled uniformly per second. The resulting sampled video sequence is: ,in Indicates the first Frame image, This represents the total number of frames currently received. Then, sampled frames are sequentially added to the array of length [number to be filled in]. The queue, when the queue is full When frames are in use, they are constructed as a time segment. And through a pre-trained spatiotemporal feature extraction network Encode the segment to obtain a feature vector that incorporates short-term spatiotemporal changes. ,in For feature dimensions.
[0058] The extracted spatiotemporal features Recorded up to the last frame of this time segment After calculating the features, the queue slides one or more frames according to a first-in, first-out (FIFO) strategy, continuing to collect new sampled frames to form the next temporal segment, thus realizing a real-time, continuous online spatiotemporal feature extraction process. Repeating the above operation yields the spatiotemporal feature sequence representation. In this process, the features of each frame depend only on historical frame information, thus satisfying the causal constraints of online processing and serving as input for subsequent action representation learning and online temporal action segmentation.
[0059] Step S2: Perform feature separation on the spatiotemporal feature sequence to obtain viewpoint-related context representation features and viewpoint-independent action representation features.
[0060] Reference Figure 2 To achieve viewpoint-robust action representation, this invention uses spatiotemporal feature sequences. The inputs are fed into the view feature learning branch and the action feature learning branch respectively, so as to learn view-related context representations and view-independent action representations simultaneously.
[0061] In the viewpoint feature learning branch, a viewpoint feature encoder is constructed using causal temporal convolution. Feature compression is performed on the input sequence to extract viewpoint-related contextual representations. ,in This represents the compressed feature dimension. A viewpoint feature decoder is then constructed using causal temporal convolution. The compressed features are restored to their original dimensions and represented as follows: ,in This represents the restored viewpoint feature sequence.
[0062] In the action feature learning branch, an action feature encoder is constructed using causal temporal convolution. The input sequence is compressed to extract viewpoint-independent action representations. Subsequently, an action feature decoder was constructed using causal temporal convolution. The compressed action representation is restored to its original feature dimension. ,in This represents the sequence of action features after recovery.
[0063] The features output from the two branches are summed element-wise to obtain the reconstructed spatiotemporal feature sequence. To ensure that key semantic information is not lost during feature separation and compression, the original spatiotemporal feature sequence... With the reconstructed feature sequence Apply reconstruction loss function , is represented as:
[0064] ;
[0065] in This represents the L2 norm.
[0066] like Figure 3 As shown, to ensure that the viewpoint feature learning branch can effectively extract viewpoint-related contextual information, the specific implementation steps of the viewpoint representation learning module are as follows:
[0067] In the viewpoint feature learning branch, to extract viewpoint-related contextual information, this invention employs an adaptive temporal feature aggregation method. For the first... Frame viewpoint features Dynamically aggregate short-term historical information to generate a viewpoint feature representation for the current frame. .
[0068] This process can be represented as:
[0069] ;
[0070] in For the first Frame and its predecessor The aggregate weights are obtained by dynamically calculating the correlation between frames. The similarity calculation function can be implemented as a lightweight feedforward network or inner product similarity calculation. By dynamically weighting and fusing short-term historical viewpoint features, a viewpoint feature representation that is more robust to noise and viewpoint perturbations is obtained.
[0071] Then, the viewpoint features of the current frame are represented. Input to view-aware embedding mapping function This function can be implemented as a two-layer linear projection and channel gating mechanism to effectively model the viewpoint dependencies between channels and enhance feature representation capabilities.
[0072] After mapping, the view classification probability of the current frame is obtained:
[0073] ;
[0074] in, The number of view categories predefined in the training set.
[0075] Based on real-world perspective tags Define the perspective classification loss:
[0076] ;
[0077] Viewpoint classification loss can help models distinguish feature differences from different shooting angles, but its limitation is that it cannot effectively characterize the continuous similarity relationship between viewpoints. For example, at 60° and 65° overhead views, although the two are of different categories, their visual features have high similarity. If only classification loss is relied upon, the model may overemphasize the differences and ignore this intrinsic connection. To address this, this invention further introduces a contrastive viewpoint discriminator. This method takes viewpoint embeddings from different videos as input and calculates the viewpoint contrast loss by constructing positive and negative sample pairs. This enhances the model's ability to perceive feature relationships between similar viewpoints while distinguishing viewpoint differences, thereby improving the continuity and robustness of viewpoint representation learning. The viewpoint contrast loss is expressed as:
[0078] ;
[0079] in, The cosine similarity function is used. For temperature coefficient, Denotes the set of positive sample pairs. This is a candidate set containing both positive and negative samples.
[0080] Finally, by jointly optimizing the view classification loss and the view contrast loss, we achieve explicit modeling of view-related contextual information, expressed as:
[0081] ;
[0082] in, The weighting coefficients are used to balance the impact of the two losses on the final loss. This joint optimization strategy can effectively guide the viewpoint feature learning branch to capture contextual information in the video that is closely related to changes in the shooting viewpoint.
[0083] like Figure 4 As shown, to ensure that the action feature learning branch can effectively extract viewpoint-independent action representation information, the specific implementation steps of the action representation learning module are as follows:
[0084] This invention introduces an action prototype representation mechanism, which interacts with video features through cross-attention to achieve unified modeling of cross-view action features, thereby generating robust view-independent action representations. The action prototype representation is as follows: ,in Indicates the first Action prototypes are trainable parameters that progressively capture key semantics of actions in real-world videos during training, enabling the modeling of fragment-level features for different chemical experimental procedures. The action prototype set... The input is fed into the self-attention mechanism module to capture the intrinsic correlations between different action prototypes, enhancing the contextual consistency of fragment-level action representations. This process is represented as:
[0085] ;
[0086] in This represents the self-attention mechanism. It represents viewpoint-independent actions. With the action prototype set updated through self-attention Perform cross-attention operations to transfer the cross-view consistency of action prototypes to frame-level features. This process is represented as:
[0087] ;in This represents the cross-attention mechanism. This operation effectively enhances the semantic consistency of different actions within frame-level features and improves the model's ability to perceive action boundaries within segments. Subsequently, the fused viewpoint-independent action representation... Input to the self-attention module to establish frame-level temporal dependencies, represented as:
[0088] Subsequently, the fused action representation is classified using a linear classification head to predict frame-level action class probabilities, denoted as:
[0089] ;
[0090] Predicted action probability distribution This reflects the model's confidence in the corresponding action category. Finally, it is based on real frame-level labels. Calculate the frame-level classification loss:
[0091] ;
[0092] in, This represents the number of action categories.
[0093] for This involves another cross-attention interaction with the action prototype representation to enhance semantic consistency between the two. This process is represented as:
[0094] Attention score during the process This describes the similarity between frame-level features and action prototypes. If the action prototypes have indeed learned cross-viewpoint consistent action semantics, then the attention scores should have a consistent distribution across different viewpoints of the same experimental process. Therefore, this invention introduces an additional cross-viewpoint consistency loss based on the attention scores, expressed as:
[0095] ;
[0096] This constraint enhances the cross-perspective robustness of action prototypes, enabling the model to learn more universal action semantics. Finally, an action-level classification loss is applied to the action prototypes output by the cross-attention mechanism to improve their discriminative ability.
[0097] ;
[0098] Therefore, the total loss of the action feature learning branch is defined as:
[0099] ;
[0100] in, The weight hyperparameter is used to balance the influence of different loss terms. Through the above joint optimization process, the action feature learning branch can effectively extract viewpoint-independent action representations and achieve robust modeling of action semantics in multi-view chemical experiment videos through prototype alignment and cross-view consistency constraints.
[0101] Step S3: Input the viewpoint-independent action representation into the online temporal segmentation network, dynamically adjust the classification probability by combining the short-term historical segmentation results, and obtain the final temporal segmentation result of the current frame.
[0102] Viewpoint-independent action features Input any online temporal action segmentation network The action classification probability distribution of the current frame is obtained as follows:
[0103] ;
[0104] To suppress over-segmentation caused by video noise, sudden changes in viewpoint, or blurred action boundaries, this invention introduces a dynamic smoothing mechanism based on short-term history during online inference. Let... This represents the classification probability distribution of the previous frame. If is the time-varying smoothing coefficient, then the dynamic correction result for the current frame is expressed as:
[0105] ;
[0106] This mechanism can effectively improve the stability and continuity of action boundaries. The classification probabilities are then dynamically smoothed and corrected. By performing the maximum probability selection, the final action segmentation result for the current frame is obtained:
[0107] ;
[0108] The results are saved to the action segmentation sequence in real time. This enables online motion segmentation of the motion state of each frame in a chemical experiment video stream.
[0109] To verify the effectiveness of the method of this invention, we conducted quantitative analysis experiments on a chemical experiment video dataset. Specifically, we constructed a video dataset for a "carbon dioxide preparation" experiment, covering four shooting methods: top-down view, front view, side view, and first-person view. In the experiment, we used top-down, first-person, and front-view videos as the training set, and side-view videos as the test set, thereby evaluating the model's generalization ability under cross-view conditions.
[0110] Experimental results are as follows Figure 5As shown, compared with representative comparative methods MS-TCN and ASFormer, the proposed method in this invention achieves significant improvements in frame-level classification accuracy (Acc), segment coherence (Edit), and action segment-level evaluation metrics (F1 score). This result demonstrates that by introducing viewpoint-robust feature separation and viewpoint-independent action representation learning, this invention effectively improves the model's action recognition and segmentation performance under different viewpoints, maintaining more stable and accurate segmentation results in real-world application scenarios.
[0111] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. The multimodal fusion described in the present invention can be extended to all modalities that can be obtained in any scenario, and is not limited to the environment and device modalities in the examples of the present invention. For those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. An online action segmentation method for chemical experiment videos based on viewpoint robust learning, characterized in that, Includes the following steps: S1. Perform frame-level sampling on the real-time chemical experiment video stream and obtain the spatiotemporal feature sequence of consecutive frames through a pre-trained visual feature extraction network. S2. Perform feature separation on the spatiotemporal feature sequence to obtain view-dependent context representation features and view-independent action representation features; In step S2, the steps of learning view-related context representations and view-independent action representations from the spatiotemporal feature sequence specifically include: S21. Input the original spatiotemporal feature sequence into the viewpoint feature learning branch and the action feature learning branch respectively, so as to learn the viewpoint-related context representation and the viewpoint-independent action representation at the same time. S22. In the viewpoint feature learning branch, the viewpoint feature encoder is used to compress the input spatiotemporal feature sequence, extract the viewpoint-related context representation, and then restore it to the original feature dimension through the viewpoint feature decoder. S23. In the action feature learning branch, the action feature encoder is used to compress the input spatiotemporal feature sequence, extract the viewpoint-independent action representation, and restore it to the original feature dimension through the action feature decoder. In step S23, the step of extracting viewpoint-independent action representations in the action feature learning branch specifically includes: S231. Establish action prototype representations to learn fragment-level action features of different operations; S232. Input the action prototype representation into the self-attention mechanism module to capture the intrinsic correlation between different action prototypes; S233. The viewpoint-independent action representation is fused with the action prototype representation updated by the self-attention mechanism module through a cross-attention operation and output as a viewpoint-independent action representation to enhance the semantic consistency of frame-level features within different actions. S234. Input the viewpoint-independent action representation output in step S233 into the self-attention module to establish frame-level temporal correlation, and apply frame-level classification loss to the output viewpoint-independent action representation. S235. Perform cross-attention operation again between the action prototype representation output in step S232 and the view-independent action representation output in step S234 to further enhance the semantic alignment effect between the action prototype and the view-independent action representation, while retaining the attention score. S236. Apply cross-view consistency alignment loss to the attention scores of the same experimental video under different viewpoints to enhance the cross-view robustness of the action prototype. S237. Apply an action-level classification loss to the action prototypes output by the cross-attention mechanism to improve the cross-view discriminativeness of the action prototypes; S24. Sum the outputs of the two branches element by element to obtain the reconstructed spatiotemporal feature sequence. Apply a reconstruction loss function between the original spatiotemporal feature sequence and the reconstructed spatiotemporal feature sequence to ensure that key spatiotemporal semantic information is preserved during feature separation and compression. S3. Input the viewpoint-independent action representation into the online temporal segmentation network, and dynamically adjust the classification probability by combining the short-term historical segmentation results to obtain the final temporal segmentation result of the current frame.
2. The online action segmentation method for chemical experiment videos based on viewpoint robust learning according to claim 1, characterized in that: In step S1, the step of extracting the spatiotemporal feature sequence from the real-time chemical experiment video stream specifically includes: S11. Perform frame-level sampling on the input video stream; S12. Several consecutive frames after sampling are combined into a time segment. The time segment is then encoded using a pre-trained spatiotemporal feature extraction network to obtain spatiotemporal features that incorporate short-term spatiotemporal changes. S13. The above-mentioned spatiotemporal features are retained and recorded in the last frame of the time sequence segment to ensure that the feature generation process satisfies causal constraints and forms a spatiotemporal feature sequence in the continuous video stream.
3. The online action segmentation method for chemical experiment videos based on viewpoint robust learning according to claim 2, characterized in that: In step S12, frame-level sampling is performed using uniform sampling.
4. The online action segmentation method for chemical experiment videos based on viewpoint robust learning according to claim 3, characterized in that: In step S22, the step of extracting view-related contextual representations in the view feature learning branch specifically includes: S221. Perform adaptive temporal feature aggregation on the input spatiotemporal feature sequence, dynamically adjust the aggregation weights according to the context information of historical frames, and generate the viewpoint feature representation of the current frame. S222. Input the view feature representation generated in step S221 into the view perception embedding mapping module to obtain the view classification probability distribution of the current frame. Calculate the view classification loss based on the real view label to constrain the model to learn discriminative view features. S223. Input the viewpoint feature representations generated from different videos into a contrastive viewpoint discriminator, and calculate the viewpoint contrast loss by constructing positive and negative sample pairs. S224. Jointly optimize the viewpoint classification loss and viewpoint contrast loss to guide the viewpoint feature learning branch to fully capture viewpoint-related contextual information in the video.
5. The online action segmentation method for chemical experiment videos based on viewpoint robust learning according to claim 4, characterized in that: The viewpoint feature decoder and action feature decoder are constructed by causal temporal convolution, respectively.
6. The online action segmentation method for chemical experiment videos based on viewpoint robust learning according to claim 1, characterized in that: In step S3, the step of obtaining the final segmentation result based on the viewpoint-independent action representation features specifically includes: S31. Input the viewpoint-independent action representation into the online temporal segmentation network to generate the action classification probability of the current frame; S32. Compare the action classification probability of the current frame generated in S31 with the classification results of the short-term history, and suppress short-term noise fluctuations through a dynamic smoothing mechanism based on the short-term history. S33. The classification probability, after being corrected by a dynamic smoothing mechanism based on short-term history, is selected with the highest probability to obtain the final action segmentation result of the current frame.