Efficient supervised action segmentation method based on frame-level and action-level double-layer time modeling

Abstract

The invention discloses an efficient supervised action segmentation method based on frame-level and action-level double-layer time modeling, which relates to the technical field of computer vision, and comprises the steps of learning a frame relationship by using a convolutional neural network, learning an action dependency relationship by using a converter, and carrying out communication between the two relationships through a cross-self-attention mechanism. A frame branch captures temporal information between frames using multiple CNN networks, an action branch initializes action flags using a multi-head cross-attention and self-attention transformer, using cross-attention to allow frame features to utilize high-level action dependencies learned by the action branch while allowing the action flags to access low-level information in the frame branch, and finally, carrying out final prediction through an average value of the features. According to the method, an action mark set with a fixed size can be efficiently learned to clearly represent an action segment, and actions and frame-level time modeling are executed in parallel, so that the method can be superior to a frame-based and two-stage method at lower calculation cost.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling. Background Technology

[0002] The goal of video action understanding is to detect, recognize, and segment human actions in videos, with wide applications in health, robotics, assistive technology, and other fields. Unlike action recognition tasks, which are used to classify actions in short video clips, action segmentation aims to divide long, unedited videos into non-overlapping action segments.

[0003] A key challenge in this task is understanding the long-term temporal dependencies between actions, as many actions can only be accurately classified within the context of other actions. Many existing action segmentation studies employ frame-based models, estimating action relationships from frame features. However, these models have an inherent disadvantage when dealing with long videos, as they must infer action relationships from tens of thousands of frames. Recent research has used transforms instead of temporal rolls to improve segmentation performance, but this significantly sacrifices efficiency. Therefore, an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling is needed to identify and segment different actions or behaviors from videos and assign the correct label or category to each action segment. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by this invention is that when processing long videos, the model must infer action relationships from tens of thousands of frames. This invention uses a two-layer temporal modeling framework to learn action tags to represent action segments, estimating temporal relationships at both the frame and action levels to achieve cross-level communication while maintaining low computational cost.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling, comprising: using a convolutional neural network to capture temporal information between frames and establishing frame branches; constructing action branches and using a transformer to learn action-level dependencies with a small number of action labels; establishing an update block and using cross-attention to allow the frame branches to utilize the high-level action dependencies learned by the action branches, while allowing the action branches to access low-level information in the frame branches; performing temporal sampling by downsampling the frame features before cross-attention and then upsampling them; and a prediction module calculating the predicted value of the action branch and the average of the predicted values ​​as the final prediction.

[0007] As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the method comprises: forming a neural network with multiple convolutional layers to extract temporal information between each frame, converting video segments into vector representations, and performing initial feature learning for frame branches; the frame branches are represented by the following formula:

[0008] ;

[0009] ;

[0010] in, These are the updated frame features; It is the initial estimate of the action probability of the frame; F1 is the frame feature, and yes and The connection is also the output of the frame branch; F0 is the original frame feature.

[0011] As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the transformer is a transformer with multi-head cross-attention and self-attention; the process of constructing the action branch includes initializing action tags using the transformer, processing a fixed-size action tag set in all videos, capturing high-level action relationships, converting action information into vector representations, and performing initial feature learning for the action branch.

[0012] The action branch is represented by the following formula:

[0013] ;

[0014] ;

[0015] Among them, among them, These are updated action branch features. It is the probability of the marked action. and A1 represents the initial estimates for the frame branch and the action branch, respectively, while A2 is the output of the action branch. These are the original action branch features.

[0016] As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the update block includes updating the action branch and performing a cross-attention operation; wherein the action branch output acts as a query, while the frame branch output acts as both the key and value, as specified in the formula:

[0017] ;

[0018] in, Indicates an updated action tag. It is the single attention map obtained; Each line represents the attention from one marker to all frames, summing to one.

[0019] Action branch scores utilize information from frame branch outputs to learn high-level action dependencies. A Transformer is used to refine the updated action branch information and predict the action category. The specific formula is as follows:

[0020] ;

[0021] ;

[0022] in, , , These are the optimized action tags.

[0023] The frame branch is updated using single-head cross-attention and refined through convolution. The specific formula is as follows:

[0024] ;

[0025] ;

[0026] ;

[0027] in, It is attention from frame to action marker; A2 is the update estimate of the frame action; A2 and F2 are the inputs to the next update block.

[0028] As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the temporal sampling includes upsampling by appropriately downsampling the frame features before cross-attention when there is a significant difference in the number of actions and frames; specifically, this involves calculating downsampled features. to replace Obtain downsampling features And as the output of cross-attention, the downsampled features are... Perform upsampling to obtain The specific formula is as follows:

[0029] ;

[0030] ;

[0031] Specifically, the frame branch output is downsampled at a non-fixed ratio, short feature segments are deleted, and then upsampled again in combination with the updated frame branch features.

[0032] As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the prediction module includes: calculating the predicted category of action tags; finding frames related to the tags using a cross-attention mechanism; segmenting the video into segments based on the action prediction of the frames; calculating a feature for each segment through average pooling; refining the feature using GRU; and performing a final prediction using the average of all features. As a preferred embodiment of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling described in this invention, the video is segmented into different segments based on the action prediction of the frames, as specified in the following formula:

[0033] ;

[0034] in, A probability distribution of actions in a single frame or a group of frames; This is the final output, representing the index of the action category predicted based on the action probability distribution.

[0035] Another objective of this invention is to provide an efficient supervised action segmentation system based on frame-level and action-level dual-layer temporal modeling. The system includes: an input module for initial feature learning of action tags and frame features using input blocks; an update module that uses cross-attention to allow frame features to utilize high-level action dependencies learned from action branches, while allowing action tags to access low-level information in frame branches; a temporal sampling module that effectively addresses the problem of significant differences between the number of actions and frames by appropriately downsampling and then upsampling the frame features before cross-attention; and a prediction module that uses the average of the predictions from the frame branches and the action branches as the final prediction.

[0036] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described above.

[0037] A computer-readable storage medium storing a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described above.

[0038] The beneficial effects of this invention are as follows: Compared to frame-based models, this method infers dependencies between actions directly from corresponding action tags, rather than calculating dependencies from long sequences of frames. Therefore, it achieves more accurate predictions with less computation. Compared to two-stage methods, this method learns action tags in parallel while learning frame features, allowing them to leverage additional information through cross-focus. It also incorporates multiple blocks for iterative refinement of predictions. This method is computationally efficient, using transformers in the action branches without incurring significant computational overhead because the number of action tags is much smaller than the number of frames. It uses temporal convolutions in the frame branches, further reducing computation, while frame features can access long-term temporal information from the action tags. This method estimates all action tags simultaneously, whereas previous works could only generate them autoregressively. This method can leverage action tags to acquire external knowledge. Some videos (e.g., instructional videos) have textual narratives that can be parsed into video transcriptions describing the action sequences in the video, which can improve accuracy with less training data. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 The flowchart shows the overall process of an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling, as provided in one embodiment of the present invention.

[0041] Figure 2 The model structure diagram of the efficient supervised action segmentation system based on frame-level and action-level dual-layer temporal modeling provided for the second embodiment of the present invention is shown. Detailed Implementation

[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0044] Example 1

[0045] Reference Figure 1 As an embodiment of the present invention, an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling is provided.

[0046] Given a test video containing T frames and pre-extracted frame-level features X (containing D feature dimensions), the goal is to predict the action labels y for all frames, where y ∈ {1, ..., A} (where A is the number of action categories). During the training phase, the video and its real action segments are provided with their label information.

[0047] Specifically, the present invention includes the following steps:

[0048] S1: Use a convolutional neural network to capture the temporal information between frames and establish frame branches.

[0049] ;

[0050] ;

[0051] in, These are the updated frame features; It is the initial estimate of the action probability of the frame; F1 is the frame feature, and yes and The connection is also the output of the frame branch; F0 is the original frame feature.

[0052] Ideally, by processing video frames using CNNs, the model can effectively capture visual features within frames and subtle changes between frames, providing rich spatiotemporal information for action recognition. This step is crucial for understanding continuous actions in videos, especially in scenes with subtle changes in motion.

[0053] S2: Construct action branches and use a transformer to learn action-level dependencies with a small number of action tags.

[0054] Specifically, a Transformer with multi-head cross-attention and self-attention is used to initialize action tags, and frame features F1 are used to learn the dependencies between tags.

[0055] ;

[0056] ;

[0057] in, These are updated action branch features. It is the probability of the marked action. and A1 represents the initial estimates for the frame branch and the action branch, respectively, while A2 is the output of the action branch.

[0058] Ideally, the introduction of a transformer enables the model to understand long-range dependencies between actions, improving the accuracy and robustness of action recognition. For complex action sequences, such as sports matches or human interaction scenarios, this step can significantly improve recognition efficiency.

[0059] S3: Establish an update block, using cross attention to allow the frame branch to utilize the high-level action dependencies learned by the action branch, while allowing the action branch to access low-level information in the frame branch.

[0060] First, cross-attention with an attention head is used to update action tags based on frame features. As a query Used as keys and values ​​to update action tags.

[0061] ;

[0062] in, Indicates an updated action tag. It is the single attention map obtained. Each row represents the attention from one marker to all frames, summing to one. It should be noted that an attention map is used because it allows the invention to obtain the alignment between action markers and frames (indicating the temporal position of the segment related to the marker).

[0063] This invention uses a Transformer with only multi-head self-attention to optimize the labeling and predict their action categories.

[0064] ;

[0065] ;

[0066] in, , , These are the optimized action tags, their probability of action category, and the output of the action branch, respectively.

[0067] The updated output A2 of the action branch, similarly uses single-head cross attention to update frame features and optimizes them through convolution.

[0068] ;

[0069] ;

[0070] ;

[0071] in, It is the attention from frame to action tag (each column sums to one). It also indicates the alignment between action tags and frames based on the latest features. This is the update estimate of the frame action. Finally, A2 and F2 are the inputs for the next update block.

[0072] Ideally, the frame branch and action branch should learn from each other. The frame branch can optimize using action dependencies, while the action branch can access low-level features in the frame branch. This mutual learning mechanism enhances the model's comprehensive understanding of the video content, especially in scenes where actions are closely related to background information.

[0073] S4: Perform temporal sampling by temporally downsampling the frame features before cross-attention and then upsampling them.

[0074] Specifically, learning cross-attention becomes challenging when there is a significant difference between the number of actions and the number of frames. Alternative strategies such as Viterbi decoding offer better performance but come at a higher cost. The temporal sampling module effectively addresses this issue by appropriately downsampling frame features temporally before performing cross-attention and then upsampling them.

[0075] Ideally, the application of temporal downsampling and upsampling reduces the computational burden when processing video while maintaining the integrity of the temporal sequence. For long videos or high frame rate videos, temporal sampling is key to improving processing speed while avoiding information loss.

[0076] This invention calculates downsampling features To replace the previous one Obtain downsampling features It is the output of cross-attention. Then, it is upsampled to obtain... The F1 score is not downsampled at a fixed ratio because this would remove features from short segments. This invention divides the video into different segments based on frame motion prediction.

[0077] ;

[0078] Then, a feature for each segment is calculated using average pooling. The obtained features are further optimized using GRU.

[0079] ;

[0080] Next, in order to upsample The features are copied. Since the upsampled features lose low-level details of the frame, they are merged with F1 using a fully connected layer, i.e.,

[0081] ;

[0082] Use this in the formula. Note that the attention maps obtained in this process are also downsampled, so upsample them for later use.

[0083] S5: The prediction module calculates the predicted value of the action branch and the average of the predicted values ​​as the final prediction.

[0084] Specifically, the predicted action category for the action tag is calculated using either the output of the frame branch or the output of the action branch, and the frame associated with the tag is determined using the final cross-attention. The final prediction is the average of these two predictions.

[0085] In summary, compared to frame-based models, this method infers dependencies between actions directly from corresponding action tags, rather than computing dependencies from long sequences of frames. Therefore, it achieves more accurate predictions with less computation. Compared to two-stage methods, this method learns action tags in parallel while learning frame features, allowing them to leverage additional information through cross-focus. It also incorporates multiple blocks for iterative refinement of predictions. This method is computationally efficient, using transformers in the action branches without incurring significant computational overhead because the number of action tags is much smaller than the number of frames. It uses temporal convolutions in the frame branches to further reduce computation, while frame features access long-term temporal information from the action tags. This method estimates all action tags simultaneously, whereas previous works could only generate them autoregressively. This method can leverage action tags to acquire external knowledge. Some videos (e.g., instructional videos) have textual narratives that can be parsed into video transcriptions describing the action sequences in the video, which can improve accuracy with less training data.

[0086] Example 2

[0087] Reference Figure 2As an embodiment of the present invention, an efficient supervised action segmentation system based on frame-level and action-level dual-layer temporal modeling is provided, including an input module that uses input blocks to perform initial feature learning for action tags and frame features; an update module that uses cross-attention to allow frame features to utilize high-level action dependencies learned from action branches, while allowing action tags to access low-level information in frame branches; a temporal sampling module that effectively addresses the problem of large differences between the number of actions and frames by appropriately downsampling the frame features before cross-attention and then upsampling them; and a prediction module that uses the average of the two predictions from frame branches and action branches as the final prediction.

[0088] Example 3

[0089] This is one embodiment of the present invention, which differs from the previous two embodiments in that:

[0090] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0092] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0093] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0094] Example 4

[0095] As an embodiment of the present invention, an efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling is provided. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0096] The proposed method was evaluated on four challenging datasets and compared with previous methods, specifically, the best frame-based and two-stage methods, ASFormer and UVAST. One-to-one matching was learned on the Breakfast, GTEA, and EgoProceL datasets using 60, 60, and 200 labels respectively, and one-to-many matching was learned on the EPIC-Kitchen dataset using 300 labels. New state-of-the-art results were achieved on all metrics across all datasets, surpassing previous state-of-the-art results on F1@50: improvements of 6.7%, 1.6%, 7.5%, and 6.1% on Breakfast (a widely used benchmark), GTEA (small-scale), EgoProceL (diverse tasks), and EPIC-Kitchen (complex long videos), respectively. More importantly, on Breakfast, BIT's inference time is 30 times faster than UVAST and 6 times faster than ASFormer.

[0097] This invention proposes an efficient supervised action segmentation method based on frame-level and action-level two-layer temporal modeling. It efficiently learns a fixed-size action tag set to explicitly represent action segments; and performs action and frame-level temporal modeling in parallel, enabling it to outperform frame-based and two-stage methods at a lower computational cost. The action tag design also allows this invention to incorporate text transcription when available, thus achieving higher segmentation accuracy even with less training data. Experimental results on multiple datasets demonstrate its superior performance in achieving accurate and cost-effective temporal supervised action segmentation tasks.

[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling, characterized in that, include: Use a convolutional neural network to capture temporal information between frames and establish frame branches; Construct action branches and use a transformer to learn action-level dependencies with action tags; An update block is established, which enables the frame branch to utilize the action dependencies learned by the action branch through cross attention, while allowing the action branch to access low-level information in the frame branch. Temporal sampling is performed by downsampling the frame features temporally before cross-attention, followed by upsampling. The prediction module calculates the predicted value of the action branch and the average of the predicted values ​​as the final prediction.

2. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 1, characterized in that: A neural network is formed by multiple convolutional layers to extract the temporal information between each frame, convert the video segments into vector representations, and perform initial feature learning for frame branches. The frame branch is represented by the following formula: F1 refine =convolution(F0); F1=concat(F1 refine ,P1); Among them, F1 refine P1 is the updated frame feature; P1 is the initial estimate of the action probability of the frame; F1 is the frame feature, and F1 is F1. refine The connection to P1 is also the output of the frame branch; F0 is the original frame feature.

3. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 2, characterized in that: The transformer is a transformer with multi-head cross-attention and self-attention; The process of constructing the action branch includes initializing action tags using the transformer, processing a fixed-size action tag set in all videos, capturing high-level action relationships, converting action information into vector representations, and learning the initial features of the action branch. The action branch is represented by the following formula: A1 refine =Transformer(A0+ρa;F1+ρf); A1=concat(A1 refine ,Pa1); Among them, A1 refine A1 is the updated action branch feature, Pa1 is the labeled action probability, ρa and ρf are the initial estimates of the frame branch and action branch respectively, A1 is the output of the action branch, and A0 is the original action branch feature.

4. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 3, characterized in that: The update block includes, Update the action branch and perform a cross-attention operation; wherein the action branch output acts as a query, while the frame branch output acts as both the key and value, as shown in the following formula: A2 attn ,Λ2=cross-attention(A1+ρa;F1+ρf); Among them, A2 attn Λ2 represents the updated action tag, and Λ2 is the single attention map obtained; each row of Λ2 represents the attention from one tag to all frames, which sums to one. Action branch scores utilize information from frame branch outputs to learn high-level action dependencies. A Transformer is used to refine the updated action branch information and predict the action category. The specific formula is as follows: A2 refine =Transformer(A2 attn +ρa); A2=concat(A2 attn ,P2); Among them, A2 refine P2 and A2 are the optimized action markers, respectively. The frame branch is updated using single-head cross-attention and refined through convolution. The specific formula is as follows: F2 attn ,Λf2=cross-attention(F1,ρf;A2,ρa); F2 refine =convolution(F2 attn ); F2=concat(F2 refine ,Pf2); Where Λf2 is the attention from frame to action tag; Pf2 is the update estimate of frame action; A2 and F2 are the inputs to the next update block.

5. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 4, characterized in that: The time sampling includes, When there is a large difference between the number of actions and frames, upsampling is performed by appropriately downsampling the frame features in time before cross attention; Specifically, the downsampling feature F1^ is calculated to replace F1, resulting in the downsampling feature F2^. attn And as the output of cross-attention, the downsampled feature F2^ attn Upsampling is performed to obtain F2 attn The specific formula is as follows: F1^ = GRU(downsampling(F1)); F2 attn =FC(F1, upsampling(F2^ attn )); Specifically, the frame branch output is downsampled at a non-fixed ratio, short feature segments are deleted, and then upsampled again in combination with the updated frame branch features.

6. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 5, characterized in that: The prediction module includes, The predicted category of the action tag is calculated, and the frame associated with the tag is found using a cross-attention mechanism. The video is segmented according to the action prediction of the frame. A feature is calculated for each segment by average pooling and refined by GRU. The final prediction is made by averaging all features.

7. The efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in claim 6, characterized in that: The video is divided into different segments based on frame motion prediction, using the following formula: y1 = argmax(Pf1); Where Pf1 represents the action probability distribution of a certain frame or a group of frames; y1 is the final output, representing the index of the action category predicted based on the action probability distribution.

8. A system employing the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in any one of claims 1 to 7, characterized in that, include: The input module uses input blocks to learn initial features for action tags and frame features; The update module uses cross-attention to allow frame features to leverage high-level action dependencies learned from action branches, while also allowing action tags to access low-level information in frame branches; The temporal sampling module effectively addresses the issue of a large discrepancy between the number of actions and frames by appropriately downsampling frame features before cross-attention and then upsampling them. The prediction module uses the average of the predictions from the frame branch and the action branch as the final prediction.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the efficient supervised action segmentation method based on frame-level and action-level dual-layer temporal modeling as described in any one of claims 1 to 7.

Images