A text-guided video temporal action segmentation method, device and storage medium

Abstract

A text-guided video temporal action segmentation method, device, and storage medium are disclosed. A text-guided video temporal action segmentation model is constructed, including a video segment encoder, a text encoder, a temporal backbone network, and a classification network. First, the video segment encoder extracts video segment features, which are then processed by the temporal backbone network to obtain temporal features containing global information. Simultaneously, the text encoder extracts text features from the linguistic descriptions of the start and end actions. The temporal features and text features are temporally modally aligned to achieve text-guided temporal action segmentation. Furthermore, to further mitigate over-segmentation caused by background interference, the temporal features undergo temporally learnable local smoothing. Finally, the obtained temporal features are classified to obtain the segmentation result. This invention enhances the constraint on video temporal features and improves segmentation performance under strong background interference by annotating the text-guided video temporal action segmentation framework.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and relates to deep learning technology and video temporal segmentation tasks. It is a method, device and storage medium for text-guided video temporal action segmentation under background interference. Background Technology

[0002] Temporal action segmentation (TAS) is a key fundamental task in video understanding. Its goal is to segment uncropped video into different time segments based on predefined action content. Through TAS, the vast amount of information distributed across different times in a video can be semantically decomposed, creating conditions for processing complex video information within limited memory. If the video is represented in time series form as S = (x... (0) x (1) , ..., x (T-1) ), where T is the video length, i.e., the number of sampling time points, and x (t) For a sampled frame or short segment from a video, it can be a pixel value or a short-time feature, t = 0, 1, ..., T-1. Then, temporal action segmentation at the segment granularity can be represented as:

[0003]

[0004] Where n is the number of action segments obtained from the segmentation, and the predicted value of the action segment is... For category predictions and duration prediction value The binary tuples are formed, i = 0, 1, ..., n-1, and each short segment x in the video is... (t) It falls into a unique, existing action segment. Therefore, temporal action segmentation can also be represented as:

[0005]

[0006] This represents the t-th short segment (i.e., x). (t) The predicted category value of ). As can be seen from the above representation, temporal action segmentation can also be regarded as a temporally dense recognition problem of short fragment sequences.

[0007] Significant progress has been made in solving this problem in recent years. These technologies can be roughly divided into several types according to their model structures: (1) Traditional temporal statistical models: random context-free grammars, hidden Markov models, semi-Markov models, etc. The modeling capabilities of neural network-based methods have significantly surpassed these models. (2) Recurrent neural networks: Gated recurrent networks (GRU), long short-term memory networks (LSTM), etc. are used to predict temporal results after encoding with image or video segment feature encoders. These methods are slightly weak in modeling broad-scale temporal information in long videos, have low tolerance for training strategies, and have low training efficiency due to their autoregressive computation. (3) Temporal convolutional networks: One-dimensional temporal convolution is used to process temporal context information, gradually expanding the receptive field to obtain dense predictions. This is also the basic architecture adopted in this invention. Its computational structure has temporal locality and time-shift invariance, which is beneficial for learning on smaller scale data and has high efficiency. (4) Transformer: The Transformer is used to model action time information by taking advantage of its natural advantage in processing sequence data. However, the characteristics of the Transformer require more data to learn better parameters. Moreover, when dealing with time segmentation problems, it often requires more special design to balance model size, efficiency and performance.

[0008] Balancing the temporal scale of action modeling with the problem of oversegmentation remains a key challenge in this field. During the execution of the same action, although the various time points within the action are semantically coherent and share a common goal, the visual features they exhibit often differ significantly. This necessitates considering more context, i.e., a broader temporal scale, when determining the action's category. Simultaneously, modeling at smaller temporal scales cannot be neglected in order to capture more action details. The oversegmentation problem refers to the algorithm incorrectly dividing a single, continuous action segment into multiple distinct action segments. This is typically caused by inadequate modeling of action continuity and context, or by the model being overly sensitive to minute changes.

[0009] Taking the action of opening a door as an example, it's impossible to determine whether the action is unlocking or locking based solely on a few frames of key turning; only the previous or subsequent door states can confirm this. Similarly, in temporal action segmentation, algorithms need to consider these video characteristics and balance modeling across broad and narrow time scales. If the model can freely exchange information at various time points (such as naive self-attention or multilayer perceptrons), it may be difficult to train or require a large dataset. Conversely, if the model only considers local information exchange (such as naive temporal convolution), it may require significant computation to model global information.

[0010] In the most commonly used public datasets for temporal action segmentation, GTEA, 50Salads, and Breakfast, the proportion of background action in the total duration is relatively small, at 23.7%, 8.88%, and 7.31%, respectively. This has led previous temporal action segmentation methods to focus more on data with weak background interference. In specific scenarios, due to longer event intervals, longer waiting times, and shorter time periods requiring attention, the proportion of background action in video data is much higher than in the above datasets. This data distribution results in greater internal differences in the performance of actions within the background time period, a higher proportion of which, and they are often adjacent to foreground actions. Therefore, such background interference can significantly interfere with the segmentation algorithm, mainly manifesting as oversegmentation. Summary of the Invention

[0011] The technical problem of this invention is to design a video temporal action segmentation framework guided by labeled text, which utilizes the text information naturally generated during annotation to enhance the constraint on the temporal features of the video, endows visual features with semantic information to enhance the model's ability to model within action segments, and at the same time alleviates the oversegmentation problem in video data with strong background interference, thereby improving segmentation performance.

[0012] The technical solution of this invention is as follows: a text-guided video temporal action segmentation method, which constructs a text-guided video temporal action segmentation model, including a video segment encoder, a text encoder, a temporal backbone network, and a classification network. First, the video segment encoder extracts segment features from each short segment sampled from a long video. Simultaneously, the text encoder extracts text features from the language descriptions of the start and end actions. Then, the temporal backbone network models global temporal information from the segment features spliced ​​along time, obtaining temporal features Z containing global temporal information. The obtained temporal features Z and text features are aligned through contrastive learning to achieve text feature-guided temporal action segmentation. To further alleviate the oversegmentation problem caused by background interference, the temporal features Z are locally smoothed, i.e., resampled temporally and smoothed in a learnable manner. Finally, the obtained aligned and smoothed temporal features are classified to obtain the segmentation result. The loss function during the training of the video temporal action segmentation model includes temporal cross-entropy loss, smoothing loss modeled by KL divergence, and modal alignment loss based on text alignment.

[0013] The present invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned video temporal action segmentation model and complete the video temporal action segmentation task on the input video.

[0014] The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program, when executed, implements the above-described video temporal action segmentation model.

[0015] This invention proposes a method for video temporal action segmentation guided by textual information. It utilizes the continuous nature of video features to construct a textual feature space for guidance, aligning video features temporally. By designing a learnable smoothing module, it effectively segments long videos temporal actions, achieving good performance. The objectives are: (1) To fully utilize the naturally generated language descriptions during dataset creation to guide the model in mapping video features to the semantic space, thereby improving segmentation performance. (2) To adaptively suppress oversegmentation when background actions constitute a large proportion of the dataset, causing strong interference. (3) To avoid problems such as excessively long training and inference times or excessive computational resource consumption due to inefficient computational structures or excessively large parameter sets.

[0016] The present invention has the following beneficial effects.

[0017] From the perspective of data utilization: Existing technical methods do not utilize the language information naturally generated during annotation. This invention uses this information to perform temporal comparative learning as a guide to improve temporal segmentation performance.

[0018] Traditional fully supervised temporal segmentation methods only utilize numerical labels of actions (one-hot encoding) for classification, lacking processing of semantic information about the actions. This invention leverages naturally generated language descriptions during dataset creation to align the temporal features of the video into the semantic feature space, thereby enhancing the model's ability to model intra-segment temporal sequences of each action segment and improving the algorithm's segmentation performance.

[0019] From the perspective of mitigating oversegmentation: Existing techniques perform poorly on video data with a high proportion of background time and abundant interference, resulting in significant oversegmentation. This invention utilizes text guidance and local smoothing to enhance modeling within action segments, preserving the importance of global temporal information while suppressing background interference, thus significantly reducing oversegmentation. This invention is further optimized for datasets with an even larger proportion of background time, demonstrating stronger resistance to such background interference compared to previous temporal action segmentation methods. This means that oversegmentation caused by interference can be significantly suppressed by this invention.

[0020] From a computational cost perspective: some existing methods are difficult to train and inefficient due to overly complex computational structures or low levels of parallelization. This invention improves upon convolutional networks as the backbone, maintaining both high training efficiency and segmentation performance. The technique of this invention achieves efficient and high-performance temporal action segmentation with low computational cost, a model parameter count of only 1.35M, and a short training time. Attached Figure Description

[0021] Figure 1 Box plots of cosine similarity distribution of features extracted from adjacent video segments for commonly used public temporal action segmentation datasets (GTEA, 50Salads, Breakfast) and chemical experiment video dataset (CO2).

[0022] Figure 2 This is a framework diagram of the text-guided video temporal action segmentation model of the present invention. Different colors in the horizontal timeline represent the time periods of each segmented action. The text encoder and the video segment encoder use a pre-trained model with frozen parameters.

[0023] Figure 3 This is a schematic diagram of the feature local smoothing module structure based on resampling in this invention.

[0024] Figure 4 This is a performance comparison of the present invention with existing time-series segmentation methods on a chemical experiment video dataset.

[0025] Figure 5 This is a visual comparison of the oversegmentation of the feature local smoothing module and the Transformer-based structural variant in this invention.

[0026] Figure 6 This is a time-axis comparison of the inference results of some samples on a chemical experiment video dataset for this invention. gt represents the label value and pd represents the predicted value. Detailed Implementation

[0027] The work of this invention is mainly divided into three parts: (1) proposing a framework for video temporal action segmentation guided by text. (2) designing an intermodal temporal alignment module, using annotations to align the features of the video and language modalities temporally with naturally generated natural language information. (3) designing a local smoothing module to further enhance the constraints of video local context modeling and alleviate the oversegmentation problem. The following is a detailed explanation of each part of this invention.

[0028] (1) Text-Guided Video Temporal Action Segmentation Framework. This invention extends the mainstream temporal action segmentation model framework of recent years to visual and linguistic modalities, enabling language to play a guiding role. In this framework, the first stage uses a video segment encoder to extract features of each short segment sampled from a long video, generally using action recognition models such as I3D, SlowFast, and VideoMAE. Considering the trade-off between computational cost and method performance, this invention directly uses a pre-trained video segment encoder. The second stage uses a temporal backbone network to extract features at a broader temporal scale. Since the video segment encoder only considers actions within short segments, the temporal backbone network needs to model global temporal information and extract features of each action segment. This stage can use structures such as Temporal Convolutional Networks (TCNs) for modeling. In the third stage, this invention aligns the extracted temporal features of each action segment with natural language descriptions, and then resamples the aligned features temporally and performs learnable smoothing. Alignment and smoothing are logically parallel. This stage adopts a relatively lightweight network structure while adding strong temporal constraints to the temporal features. The final temporal features can be used to obtain the segmentation results from the classification.

[0029] (2) Temporal Alignment Between Modalities. In typical temporal action segmentation methods, video segment sampling is relatively dense. Methods such as MSTCN, UVAST, and ASFormer use a sampling rate of 15 frames per second, while this invention uses 5 frames per second. In typical videos, this sampling rate results in minimal differences between adjacent video segments, leading to similar segment features obtained by the encoder. Therefore, this invention assumes that the features extracted from adjacent video segments by an ideal temporal action segmentation framework are similar. In this embodiment, VideoMAE is used to extract features from a chemical experiment dataset and the cosine similarity of adjacent segment features is calculated as follows: Figure 1 The diagram demonstrates the validity of this hypothesis. In existing temporal action segmentation tasks, the training dataset requires manual annotation of the start and end times of each action. When manually annotating video datasets, precise linguistic descriptions of action boundaries are used to ensure consistency between annotations. This invention utilizes these descriptions, directly employing existing linguistic definitions from the annotation process to constrain the start and end regions of each action segment, aligning these features to the text feature space obtained by the language encoder. The alignment network used in this invention is a temporal feedforward network, whose continuity ensures that the temporal features between the start and end of an action segment gradually change between these two regions. This invention uses a contrastive learning method to align modal features, treating adjacent temporal features as their respective data augmentations, thus eliminating the need for further data annotation.

[0030] (3) Resampling and smoothing of temporal features. To further suppress overfitting caused by a large amount of interference in video actions, this invention designs a learnable resampling and smoothing module as a regularization, so that the temporal features entering the classification at the end of the network better satisfy the above continuity, thereby reducing the impact of interference. Its structure is as follows: Figure 3 As shown, this module does not directly and mechanically smooth the data using filters or threshold-related penalty terms. Instead, it resamples the temporal features in overlapping time windows. Each sampled time window is independently fused, i.e., dimensionality reduction in the time dimension and enhancement of information exchange among local features through a linear layer. After independent normalization, these features are aggregated back to the original dimension by means of the inverse resampling operation, thus achieving stronger constraints on the temporal context.

[0031] The implementation of the present invention is illustrated below through specific examples. A video segment encoder extracts segment features from each short segment sampled in a long video, and concatenates them along the time axis to obtain the features of the entire video. The resulting video feature sequence is denoted as... D is the feature dimension. In the training set, the label value sequence for action segmentation is y = (y (0) y (1) , ..., y (T-1) The predicted value sequence is: That is, each short segment corresponds to a label y. (t) The predicted value calculated by a video temporal action segmentation model

[0032] Dataset: The dataset (CO2) of this invention contains 37 hours of videos with a total effective length, depicting experiments conducted by researchers to produce carbon dioxide from hydrochloric acid and limestone using a specific method. The dataset contains 220 video sets, with individual video lengths ranging from 5 to 30 minutes. Due to the nature of chemical experiments, the background motion in this dataset accounts for a significantly higher proportion (62.7%) compared to several publicly available temporal motion segmentation datasets. This includes a large number of semantically ambiguous and cluttered background motions, which sometimes exhibit short-lived visual features similar to key actions.

[0033] like Figure 2As shown, the text-guided video temporal action segmentation model of the present invention includes a video segment encoder, a text encoder, a temporal backbone network, and a classification network. First, the video segment encoder extracts features of each short segment sampled from a long video. Simultaneously, the text encoder extracts text features from the language descriptions of the start and end actions. Then, the temporal backbone network models the global temporal information of each short segment feature spliced ​​along time, obtaining a temporal feature Z containing global temporal information. Modality alignment and feature smoothing are performed on the temporal feature Z in parallel. The temporal feature Z and the text feature are temporally modally aligned through contrastive learning to achieve text feature-guided temporal action segmentation. At the same time, in order to further alleviate the oversegmentation problem caused by background interference, the temporal feature Z is subjected to temporally learnable local smoothing. Finally, the obtained temporal feature is classified to obtain the segmentation result.

[0034] Temporal Backbone Network: This invention utilizes a multi-stage temporal convolutional network to achieve video feature extraction over a longer time scale. The network employs numerous dilated convolution operations, enabling it to rapidly expand the temporal receptive field with relatively low computational cost. After passing through the temporal backbone network, video features X are transformed into lower-dimensional temporal features containing global information. The temporal feature at time point t is represented as z. (t) C represents the feature dimension.

[0035] Temporal alignment of representations: Aligning temporal features with textual features. If the textual feature representations corresponding to the action start definition and action termination definition in the aligned multimodal representation space are g(v... s ) and g(v t Let g be a text encoder, then as the action progresses, the video segment features z... (t) It should gradually move from g(v) s continuously move to g(v) near ) t In an ideal representation space, since the features of the two modalities are already aligned, the relationship f with the text encoder can be further given for the known linguistic descriptions at the start and end of each action:

[0036]

[0037] This invention uses a point-in-time feedforward network to achieve the above constraints:

[0038] z (t) =TFFN(f(x) (t) ))

[0039] x (t) Let S be the feature at time point t in the video feature S.

[0040] At this point in time, the feedforward network mainly consists of two linear layers:

[0041] TFFN(x)=max(0,xW1+b1)W2+b2,

[0042] Where W1, b1, W2, and b2 are the parameters of two linear layers. To extend the structured embedding representation to temporal sequences, this invention views the process within each continuous action segment as a relationship from start to finish, where the change in temporal features is the smooth transition of an action segment from start to finish. In the linguistic description of action boundaries in the dataset, the most important aspect is the description of the distinctive visual features. If the criteria for determining the start and end of an action segment are similar, then they should also be similar in the feature space, and the above mapping method can reflect the continuous change process in the middle of the modeling of the two endpoints. This invention applies the above constraints to the start and end points of each non-background action to map the temporal features of each action segment to the region between two natural language descriptions, i.e., the alignment of temporal sequence with text. Simultaneously, this relatively simple model designed in this invention can balance the variability of temporal sequence within the action and its resistance to noise.

[0043] This invention uses contrastive learning to align the features of the temporal backbone network and the text encoder. A target encoder is used on the visual side to bring similar action segments closer together in the embedding space, while dissimilar action segments are spaced further apart. This improves the accuracy and continuity of temporal action segmentation and better handles intra-action processes and inter-action variations, thus suppressing over-segmentation. When constructing sample pairs, background actions and all language description embeddings form negative sample pairs, while the start and end phases of each key action and their corresponding descriptions form positive sample pairs. For the intermediate processes of an action segment, this invention assumes that the feedforward network at these time points can learn process features from endpoint features, therefore no further constraints are added to this contrastive learning. Since the start and end samples for each action are too few, and to support the required sample size for contrastive learning, this invention selects portions with a duration of α at the beginning and end of each action segment for contrastive learning. For example, if the action segment duration is S, αS of the video portion is selected from the starting frame as the starting portion, forming a positive sample pair with the text for contrastive learning; the same applies to the end.

[0044] Feature local smoothing: such as Figure 3 As shown, the temporal feature Z is first resampled, and the sampling window length is denoted as T. w n are obtained by equally upsampling a video of length T. w One window:

[0045]

[0046] w hThe window feature is a portion of the temporal feature Z obtained by temporal upsampling, corresponding to a time window. Each subsequent w... h Processed independently, firstly, the time dimension is reduced to T′. w <T w After normalization, the data is projected back to the original time series dimension for time series fusion.

[0047]

[0048] in Indicates T′ w Projection transformation back to T w , T represents w Dimensionality reduction transformation to T′ w s h Right now Figure 3 Intermediate quantities after time-series fusion.

[0049] The transformation of Proj is expressed as:

[0050] Proj d，d′ (x d )=σ(x d W d×d′ +b d′ ).

[0051] The input of d dimensions is about to be x d The transformation to d′ dimension is performed using the weight matrix W and the bias vector b, along with the projection layer of the activation function σ. The subscripts of W and b indicate their dimensional shape.

[0052] The present invention uses a linear projection layer for local smoothing to ensure that within window w h Each temporal feature within a given window can be integrated with all temporal information within that window, meaning its receptive field in this module can cover the entire window. Temporal dimensionality reduction aims to further approximate action information at similar times while reducing computational cost. The normalization operation Norm uses instance normalization, which takes into account the various windows, i.e., n... w The dimensions remain independent, and since each window needs to be aggregated in subsequent operations, information exchange between windows should not be introduced at present. Therefore, normalization is performed independently on each window in this way.

[0053] The s obtained above h By aggregating fragments, the original temporal dimensions are restored through the inverse operation of resampling:

[0054]

[0055] This leads to the dense reasoning result.

[0056] From a data augmentation perspective, the temporal patterns contained in the features Z obtained by the temporal backbone network are equivalent to generating diverse time periods in the time dimension through resampling. Each window independently performs local temporal information fusion processing, which is equivalent to completing an automatic temporal data augmentation. The resulting augmented views are then subjected to fragment aggregation to strengthen the constraint on continuity and suppress misjudgments in individual views caused by improper modeling, thereby satisfying the continuity assumption mentioned above.

[0057] The training of the video temporal action segmentation model of this invention includes temporal cross-entropy loss, smoothing loss based on KL divergence modeling, and modal alignment loss based on text alignment, as detailed below.

[0058] Calculate the cross-entropy loss at each time point:

[0059]

[0060] Where p(y) (t) This represents the distribution of action label values. This represents the normalized probability distribution of the model output. To further enhance the model's robustness and training stability against large feature changes over short periods, a smoothing loss term using KL divergence modeling is needed to measure the change in normalized probability between adjacent time points:

[0061]

[0062] Use the shear function to limit the range of probability differences.

[0063]

[0064] Where the shearing function Where L and H are the upper and lower limits, respectively, and δ is a certain value of the limit, which is a hyperparameter.

[0065] For a single video sample, the contrastive loss of visual text information during temporal alignment is:

[0066]

[0067] The similarity function z q and v q Each sample pool contains video features and action language definitions for a single video sample. The sample pool includes the video features and action language definitions for each time point in the current video, the start and end portions of each action segment (occupying the segment duration α), and the corresponding segment features and language definitions for each time point of the background action. q It is derived from the temporal feature Z by sampling over time, v q It is z qThe language definitions at each point in time included, v (t) Let t be the language definition corresponding to the t-th time point, m be the sample number in the sample pool, M be the sample pool capacity, and τ be the temperature coefficient.

[0068] The model's loss function is a weighted sum of the above losses:

[0069]

[0070] To save computational costs, this invention does not update any parameters of the pre-trained video encoder and language encoder. All other parts of the model do not require pre-training and are optimized directly from a randomly initialized state.

[0071] Performance comparison of this invention with other major temporal action segmentation methods on carbon dioxide video datasets is as follows: Figure 4 Thanks to the incorporation of textual information and the more comprehensive utilization of temporal information, this invention surpasses previous methods in most key performance indicators. Simultaneously, due to the lightweight nature of each module, its computational efficiency remains high. While a small improvement is achieved in the naive frame-by-frame accuracy (acc), more significant improvements are made in the edit and F1 scores, which are more sensitive to the completeness and order of action segmentation. This demonstrates that this invention can overcome the interference of background actions in the dataset and significantly reduce over-segmentation.

[0072] To illustrate the effectiveness of the temporal resampling smoothing module, four different variants of the action temporal segmentation dense prediction model based on the Transformer encoder are constructed and compared. The inference results on the first sample of the GTEA dataset are visualized on the timeline as follows. Figure 5 As shown, different colors represent the time periods of different actions obtained from segmentation. The four methods used are: (1) Directly using linear layers to reduce the dimensionality of video segment features and then inputting the entire sequence into a Transformer encoder, followed by low-pass filtering post-processing. (2) Using a multi-layered naive temporal convolutional network to extract features with strong temporal position correlation, inputting the entire sequence into a Transformer encoder, and then using low-pass filtering post-processing. (3) A multi-scale feature Transformer encoder composed of jumps and downsampling / upsampling, with a structure similar to U-Net. (4) Inputting the entire sequence of video segment features into a Transformer encoder, and then applying the temporal resampling smoothing module of this invention. Figure 5As can be seen, despite using low-pass filtering for post-processing, the first two structures still exhibit severe oscillations, resulting in significant oversegmentation. The U-shaped jumper method can quickly expand the receptive field of the convolution while maintaining segmentation resolution, but it still produces many misclassifications. The temporal sampling smoothing method in this invention can essentially eliminate the oversegmentation problem and learns the prior order of each action, achieving better performance.

[0073] The timeline visualization of some inference results from this invention on a video dataset of chemical experiments in carbon dioxide production (CO2) is shown below. Figure 6 As shown in the figure, the horizontal axis represents the time axis, and the vertical axis represents the sample. Every two rows of the time axis represent the inference result pd and the label value gt for the same sample. Each action is represented by a different color, and the background is represented by white. A comparison of the inference results and label values ​​shows that the actions segmented by this invention are generally complete and accurate, even distinguishing a small number of closely adjacent repetitive actions. The deviations mainly occur at the boundary between key actions and the background, presumably due to the ambiguity of the background. In summary, this invention can accurately capture the main features and temporal relationships of each action.

Claims

1. A text-guided video temporal action segmentation method, characterized by: A text-guided video temporal action segmentation model is constructed, comprising a video segment encoder, a text encoder, a temporal backbone network, and a classification network. First, the video segment encoder extracts segment features from each sampled segment in the video. Simultaneously, the text encoder extracts text features from the linguistic descriptions of the start and end actions. Then, the temporal backbone network models global temporal information from the segment features concatenated along time, resulting in temporal features incorporating global temporal information. The obtained time series features By comparing text features with textual features and performing text alignment through contrastive learning, the segmentation of temporal actions is guided by textual features. Furthermore, to further mitigate oversegmentation caused by background interference, the temporal features are... Then, local smoothing is performed, that is, temporal resampling and learnable smoothing are performed. Finally, the aligned and smoothed temporal features are classified to obtain the segmentation result. The loss function during the training of the video temporal action segmentation model includes temporal cross-entropy loss, smoothing loss modeled by KL divergence, and modal alignment loss based on text alignment. Among them, time series features Alignment with text features specifically refers to: Action start definition Action termination definition The text features are represented as follows: and In the corresponding visual feature space, it represents the initial action. The corresponding features move continuously to the termination action. Corresponding features , These represent the start and end times of the action, respectively, indicating a correspondence between the text modality and the visual modality. ; A feedforward network at a single time point is used to extract features from video segments. Correspondence Constraints Temporal characteristics No. Temporal characteristics at each time point The time-point feedforward network consists of two linear layers: in , , , For the parameters of two linear layers; Contrastive learning is used to align the temporal features of the temporal backbone network based on the aforementioned relationships. Based on the text features of the text encoder, similar action segments are brought closer together in the embedding space, while dissimilar action segments are separated. Text features guide the segmentation of temporal actions to improve the accuracy and continuity of temporal action segmentation. Feature local smoothing specifically refers to: First, the temporal characteristics... Perform resampling, and record the sampling window length as... Temporal characteristics The feature dimension is With a fixed sampling interval, samples were taken evenly over a length of... Video upsampling One window: Next, we'll work on each window. Processed independently, firstly, the time sequence is reduced to... , After normalization, the intermediate quantity is projected back to the original temporal dimension to obtain the temporal fusion intermediate quantity. : in, Dimensional transformation is represented as: Soon Dimensional input Transform to Dimension, using weight matrix and bias vector With activation function Transform the projection layer; The windows obtained By aggregating fragments, the original temporal dimensions are restored through the inverse operation of resampling: Obtain dense reasoning results That is, the prediction of action segmentation.

2. The text-guided video temporal action segmentation method according to claim 1, characterized in that: The temporal backbone network is a multi-stage temporal convolutional network that uses dilated convolution operations to expand the temporal receptive field, transforming time-series spliced ​​fragment features into temporal features containing global information. .

3. The text-guided video temporal action segmentation method according to claim 1, characterized in that: For text alignment training, when constructing sample pairs, the background action and all language description embeddings form negative sample pairs, and the start and end phases of each key action and the corresponding text description form positive sample pairs. The duration of each action segment is determined by the proportion of time taken in the start and end frames. These components together constitute the start and end phases of comparative learning, supporting the sample size required for comparative learning.

4. The text-guided video temporal action segmentation method according to claim 1, characterized in that the loss function during the training of the video temporal action segmentation model is specifically as follows: Calculate the cross-entropy loss at each time point: in This indicates the distribution of action label values. This represents the normalized probability distribution of the model output; The smoothing loss modeled using KL divergence measures the change in normalized probability between adjacent time points: And use a shearing function to limit the range of probability differences: Where the shearing function in and These are the upper and lower limits, respectively. The value is one of the upper and lower limits, and it is a hyperparameter. For a single video sample, the contrastive loss of visual text information during temporal alignment is: The similarity function , and Each sample pool consists of video features and action language definitions for a single video sample, including the start and end times of each action segment in the current video at various time points, representing the duration of each action segment. The segment features and language definitions corresponding to each point in time of the background action. For the first The language definition corresponding to each point in time. The sample number in the sample pool. For the sample pool capacity, Temperature coefficient; The loss function for the video temporal action segmentation model is as follows: , To correspond to the weights of the loss function, the video temporal action segmentation model does not update any parameters of the pre-trained video segment encoder and text encoder during training.

5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the video temporal action segmentation model according to any one of claims 1-4, and to complete the video temporal action segmentation task on the input video.

6. A computer-readable storage medium, characterized in that... The computer-readable storage medium stores at least one instruction or at least one program, which, when executed, implements the video temporal action segmentation model as described in any one of claims 1-4.

Images