Video temporal action segmentation method based on audio and video bimodal feature fusion

Abstract

The present invention provides a video time-domain action segmentation method based on the fusion of audio and video bimodal features. A video time-domain action segmentation network model is designed, audio features are introduced into the video time-domain action segmentation task, the natural waveform features of audio and video features are used to regress the action boundary probability, and a multi-stage audio-visual bimodal cascade network is designed to capture the boundary information of the video and refine the network's frame-by-frame action prediction. At the same time, an audio-visual bimodal boundary regression module is designed to calculate audio features and video features to obtain the action boundary probability and refine the action classification results of the multi-stage audio-visual bimodal cascade network, so as to capture reliable video clips and correct video action classification, thereby alleviating the boundary fuzziness problem during video segmentation and improving the video action segmentation effect.

Description

Technical Field

[0001] The present invention relates to the technical field of video action segmentation, and in particular to a video time-domain action segmentation method based on audio and video dual-modal feature fusion. Background Art

[0002] With the successful application of deep learning in the classification of short clipped videos, temporal action segmentation and action classification of unedited long videos have received increasing attention and have become a hot research direction in video content understanding.

[0003] The task of video temporal action segmentation is to classify the action category to which each frame in an untrimmed video belongs. The question that needs to be answered is: which video frames belong to which action category. The video temporal action segmentation method can reflect the intelligent and automated analysis process of untrimmed videos, help divide multiple action segments in the video, and perform fine-grained analysis of the actions in the video to improve the efficiency of action understanding. The video temporal action segmentation method can be used in application scenarios with continuous actions, such as production line assembly, human-computer interaction, teaching videos, and the actual operation procedures of medical personnel. At the same time, it also plays a significant role in promoting the modernization, intelligence, and automation of video processing technology. Therefore, the video temporal action segmentation method has extremely important application value in the field of video intelligent and automated processing.

[0004] Currently, the most advanced video temporal action segmentation methods are implemented using multi-layer temporal convolution and temporal pooling. These methods can capture the temporal correlation of videos, but almost all of them have problems of over-segmentation and blurred boundaries.

[0005] In summary, there is an urgent need for a video temporal action segmentation method based on audio and video bimodal feature fusion to solve the problems existing in the existing technology. Summary of the Invention

[0006] The present invention aims to provide a method for video temporal action segmentation based on audio and video bimodal feature fusion, aiming to solve the problems of over-segmentation and blurred boundaries in the existing technology. The specific technical solution is as follows:

[0007] The video temporal action segmentation method based on audio and video bimodal feature fusion includes the following steps:

[0008] S1: Extract video and audio features of multimedia data;

[0009] S2: Establish a video temporal action segmentation network model and output a video temporal action segmentation classification result based on the video temporal action segmentation network model. The video temporal action segmentation network model includes a multi-stage audio-visual bimodal cascade network and an audio-visual bimodal boundary regression module. The specific steps include:

[0010] S2-1: Integrate the audio features of multimedia data into the video features to obtain the fused audio-visual bimodal features, which are then input into a multi-stage audio-visual multimodal cascade network.

[0011] S2-2: Based on audio and video features, an audio-visual bimodal boundary smoothing operator is generated through the audio-visual bimodal boundary regression module. The audio-visual bimodal boundary smoothing operator is used to smooth the frame confidence scores of each cascade stage in the multi-stage audio-visual multimodal cascade network and generate self-attention weights.

[0012] S2-3: The frame confidence scores of each cascade stage in the multi-stage audio-visual multimodal cascade network are weighted and summed according to the self-attention weight matrix, and input into the fusion stage to obtain the fusion classification score. The video temporal action segmentation classification result is output based on the fusion classification score.

[0013] Preferably, in step S2-2, the expression for smoothing the frame confidence score of each cascade stage in the multi-stage audiovisual multimodal cascade network by using the audiovisual bimodal boundary smoothing operator is:

[0014]

[0015] b′=(b video +βb audio ) / 2;

[0016] in, is the frame confidence score predicted by the network after adding the smoothing operator; is the frame confidence score; s is the direction of pooling; β is the model hyperparameter that determines the contribution of different features; L is the length, and the length of the pooling window is 2L+1; is the frame confidence score of the frame in the pooling window; α is the decay rate of the self-attention weight; j is the number of elements taken in each direction in the pooling window; b′ t+s·j is the action boundary probability within the pooling window; att is the self-attention adjustment unit; b′ is the fusion boundary probability; b video is the video boundary probability; b audio is the audio boundary probability.

[0017] Preferably, the expression of the self-attention weight in step S2-2 is:

[0018]

[0019] in, is the self-attention weight of each cascade stage in the multi-stage audio-visual multimodal cascade network; e is a natural constant; ρ is the set frame confidence score threshold; is the frame confidence score at time t and stage j′.

[0020] Preferably, the expression of the fusion classification score in step S2-3 is:

[0021]

[0022] in, is the fusion classification score.

[0023] Preferably, the step S2-3 further comprises the following step: smoothing the fusion classification score by using an audiovisual bimodal boundary smoothing operator, which is expressed as follows:

[0024]

[0025] b′=(b video +βb audio ) / 2;

[0026] in, It is the smoothed fusion classification score predicted by the network after adding the smoothing operator.

[0027] Preferably, the loss distribution on the frame in the cascade stage is also included, and the expression is as follows:

[0028]

[0029] in, is the loss distribution on the frame in the cascade stage; y t，c is the predicted probability of category C at time t; t is a certain time;

[0030] And the classification loss on the frame in the fusion stage is expressed as follows:

[0031]

[0032] in, is the classification loss; T is the total length of multimedia data.

[0033] Preferably, the smoothing loss when smoothing the frame confidence score and the fusion classification score in the cascade stage is smoothed by the audiovisual bimodal boundary smoothing operator, and its expression is as follows:

[0034]

[0035]

[0036] Δ t，c =|logy t，c -logy t-1，c |;

[0037] in, is the smoothing loss; Cn is the total number of categories; σ is the standard deviation; x t is the frame confidence score at time t; is the square of the logarithmic difference between the confidence scores of the two frames before and after; Δ t，c is the logarithmic difference of the confidence scores of the two frames before and after; τ is the threshold.

[0038] Preferably, the final loss function of the cascade stage is expressed as:

[0039]

[0040] Among them, λ is a model hyperparameter that determines the contribution of different losses; is the final loss function of the cascade stage;

[0041] The final loss function of the fusion stage is expressed as:

[0042]

[0043] in, is the final loss function in the fusion stage.

[0044] Preferably, the expression for minimizing the sum of losses of all cascade stages and fusion stages is:

[0045]

[0046] in, is the sum of the minimized losses of all cascade stages and fusion stages.

[0047] Preferably, a loss function for audiovisual bimodal boundary regression is also included, and its expression is as follows:

[0048]

[0049] Where N is the number of prediction stages; is the loss function for audiovisual bimodal boundary regression; is the cross entropy loss at each stage of boundary regression.

[0050] The application of the technical solution of the present invention has the following beneficial effects:

[0051] (1) The patent of this invention proposes a video time-domain action segmentation method based on the fusion of audio and video bimodal features, designs a video time-domain action segmentation network model, introduces audio features into the video time-domain action segmentation task (that is, integrates the audio features of multimedia data into the video features to obtain the fused audio-visual bimodal features, and inputs them into the multi-stage audio-visual multimodal cascade network of the video time-domain action segmentation network model), uses the natural waveform features of audio and video features to regress the action boundary probability, and designs a multi-stage audio-visual bimodal cascade network to capture the boundary information of the video and refine the network's frame-by-frame action prediction; at the same time, an audio-visual bimodal boundary regression module is designed to calculate the audio features and video features to obtain the action boundary probability and refine the action classification results of the above multi-stage audio-visual bimodal cascade network, capture reliable video clips and correct video action classification, so as to alleviate the boundary fuzzy problem during video segmentation and improve the video action segmentation effect.

[0052] (2) The present invention also sets the loss function of minimizing the sum of losses in all cascade stages and fusion stages and the loss function of audio-visual bimodal boundary regression, so that in the video time-domain action segmentation network model provided by the present invention, after each batch of training data is fed into the model, the predicted value is output through forward propagation, and then the loss function calculates the difference between the predicted value and the true value, that is, the loss value. After obtaining the loss value, the model adjusts the various parameters in the video time-domain action segmentation network model through back propagation to reduce the loss between the true value and the predicted value, so that the predicted value generated by the model is closer to the true value, thereby achieving the purpose of learning.

[0053] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 This is a schematic diagram of the overall process of the preferred embodiment 1 of the present invention;

[0056] Figure 2 yes Figure 1 Schematic diagram of the process of extracting audio features and video features;

[0057] Figure 3 This is a schematic diagram comparing the boundary smoothing operator based on the self-attention mechanism and the boundary smoothing operator;

[0058] Figure 4This is a schematic diagram of the multi-stage structure of audio boundary regression;

[0059] Figure 5 This is the qualitative result of the temporal action segmentation task on the EPIC-KITCHENS dataset; DETAILED DESCRIPTION

[0060] To facilitate understanding of the present invention, the present invention will be described more fully below, along with preferred embodiments thereof. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present invention.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0062] Example 1:

[0063] See also Figure 1 , a video temporal action segmentation method based on audio and video bimodal feature fusion, comprising the following steps:

[0064] S1: Extract video and audio features of multimedia data, see Figure 2 ;

[0065] Specifically:

[0066] (1) Using TSN to extract video features, including:

[0067] The ffmpeg tool is used to extract RGB frames from the mp4 files in the video dataset. A parallel environment is established to quickly and parallelly extract RGB features of the video RGB frames. The input of the TSN model is all RGB frames of a video (multimedia data), and the output feature size is T x1024, where T is the number of video frames and 1024 is the dimension of the feature. Finally, the video feature set of the video dataset is extracted.

[0068] (2) Using VGGish to extract audio features, including:

[0069] Use the ffmpeg tool to extract wav and acc files from the mp4 files in the video dataset; establish a parallel environment to achieve parallel and fast extraction of audio features of the video; the VGGish model is pre-trained on AudioSet, and the extracted features come from the activated pre-classification layer; the VGGish model inputs all the audio files of a video (multimedia data) and outputs a feature dimension of 128 dimensions; finally, the audio feature set of the video dataset is extracted.

[0070] Taking the EPIC-KITCHENS video dataset as an example, this dataset contains 633 videos totaling 1866.4GB. Using the above method, we extracted 80.2GB of video features and 0.2GB of audio features. The significant disparity in feature set size also reveals the information gap between video and audio features. Therefore, when utilizing audio and video features, we only use audio features as auxiliary information to complement the video features, primarily for learning boundaries and participating in some action classification tasks.

[0071] The video and audio features extracted from the EPIC-KITCHENS dataset are shown in Table 1.

[0072] Table 1 Video and audio features in the EPIC-KITCHENS dataset

[0073] #Hour #Action class #Number #Dimension #Size(GB) EPIC-KITCHENS 100 125 700 - 1866.4 RGB features - - 633 1024 80.2 Audiofeatures - - 633 128 0.2

[0074] S2: Establish a video temporal action segmentation network model and output a video temporal action segmentation classification result based on the video temporal action segmentation network model. The video temporal action segmentation network model includes a multi-stage audio-visual bimodal cascade network and an audio-visual bimodal boundary regression module. The specific steps include:

[0075] S2-1: Integrate the audio features of multimedia data into the video features to obtain the fused 1152-dimensional audio-visual bimodal features, which are input into the multi-stage audio-visual multimodal cascade network;

[0076] S2-2: Based on the audio and video features, the audio-visual bimodal boundary regression module generates an audio-visual bimodal boundary smoothing operator. The specific steps include:

[0077] (1) Video boundary regression

[0078] For the boundary information of video features in the video dataset, the temporal evaluation module (TEM) in the BSN model is used to regress the video boundary probability and fused it with the audio boundary probability generated by audio boundary regression to generate an audio-visual bimodal boundary smoothing operator.

[0079] (2) Audio boundary regression

[0080] For the boundary information of audio features in video datasets, a time-domain convolutional network with an expanded residual layer is designed and expanded into a multi-stage structure. This multi-stage structure is used for audio boundary regression to generate audio boundary probabilities. In this embodiment, the multi-stage structure is described in detail in the following sections. Figure 4 As shown, each stage consists of a temporal convolution with a kernel size of 1 and 64 filters, 10 dilated residual convolutions, and another temporal convolution for reducing the feature dimension to the number of action classes. The dilation rate of each dilated convolution doubles with the number of layers, and the number of stages is set to 3 after the initial prediction layer.

[0081] (3) Fusion of dual-modal boundary probability

[0082] The video boundary probability and the audio boundary probability are fused and input into the self-attention adjustment unit, and the boundary probability is mapped to a new interval to generate an audio-visual bimodal boundary smoothing operator that integrates the action boundary information of the video modality and the natural waveform features of the audio features. Figure 3 (a is the audiovisual bimodal boundary smoothing operator, and b is the ordinary boundary smoothing operator). It can be seen that the audiovisual bimodal boundary smoothing operator can use the local pooling window centered on the current frame, and calculate the self-attention weights in the left and right directions from the center, adaptively smoothing the output results of the multi-stage audiovisual bimodal cascade network (compared to the boundary smoothing operator, the audiovisual bimodal boundary smoothing operator in this application can adaptively reduce the weight when encountering a boundary; when a frame that is not near the boundary but has a very large impact on the network action segmentation result appears in the pooling window, it can also adaptively increase the weight to capture more effective information).

[0083] The frame confidence score of each cascade stage in the multi-stage audiovisual multimodal cascade network is then smoothed by the audiovisual bimodal boundary smoothing operator, and the expression is as follows:

[0084]

[0085] b′=(b video +βb audio ) / 2;

[0086] in, is the frame confidence score predicted by the network after adding the smoothing operator; is the frame confidence score at time t for each cascade stage; s is the direction of pooling; β is the model hyperparameter that determines the contribution of different features; L is the length, and the length of the pooling window is 2L+1; is the frame confidence score of the frame in the pooling window; α is the decay rate of the self-attention weight; j is the number of elements taken in each direction in the pooling window; b′ t+s·j is the action boundary probability within the pooling window; att is the self-attention adjustment unit; b is the fusion boundary probability; bvideo is the video boundary probability; b audio is the audio boundary probability;

[0087] Because the weights of all cascade stages are determined by the frame confidence score and weight of the previous cascade stage, the self-attention weight is generated, and the expression is:

[0088]

[0089] in, is the self-attention weight of each cascade stage in the multi-stage audio-visual multimodal cascade network; e is a natural constant; ρ is the set frame confidence score threshold; is the frame confidence score at time t and stage j′;

[0090] S2-3: The frame confidence scores of each cascade stage in the multi-stage audio-visual multimodal cascade network are weighted and summed according to the self-attention weight matrix, and input into the fusion stage to obtain the fusion classification score; the expression of the fusion classification score is:

[0091]

[0092] in, is the fusion classification score; fusion classification score It can adaptively combine the outputs of each frame at different cascade stages and pass them to the fusion stage to produce the final prediction of the stage cascade;

[0093] The audiovisual bimodal boundary smoothing operator is used to smooth the fusion classification score to further reduce the boundary blur problem. The video temporal action segmentation classification result is output based on the smoothed fusion classification score. The smoothed fusion classification score expression is as follows:

[0094]

[0095] b′=(b video +βb audio ) / 2;

[0096] in, It is the smoothed fusion classification score predicted by the network after adding the smoothing operator.

[0097] The audiovisual bimodal boundary-aware video temporal action segmentation algorithm takes video and audio features as input and outputs the video action segmentation result. The pseudocode for implementing this algorithm is shown in Table 2.

[0098] Table 2 Audiovisual dual-modal boundary-aware video temporal action segmentation algorithm

[0099]

[0100]

[0101] In this embodiment, in order to ensure that the model training tends to stable convergence, a multi-stage audio-visual bimodal cascade network structure and a loss function of the audio-visual bimodal boundary regression module are designed.

[0102] For the multi-stage audio-visual multimodal cascade network structure, loss functions are designed for all cascade stages and fusion stages, including classification loss and smoothing loss.

[0103] The loss distribution of the cascade stage on the frame is expressed as follows:

[0104]

[0105] in, is the loss distribution on the frame in the cascade stage; y t，c is the predicted probability of category c at time t; t is a certain time;

[0106] And the classification loss on the frame in the fusion stage is expressed as follows:

[0107]

[0108] in, is the classification loss; T is the total duration of the multimedia data; since the frequency of each action category is different in real datasets, it will lead to serious imbalance during training. If there are too many negative samples, the loss value will be too large, so that the model will unconsciously ignore the loss value of positive samples, which is not conducive to convergence. A large gap between the number of positive samples will also lead to serious imbalance during training. Therefore, the classification loss on the frame in the fusion stage is A weight is applied to each action category in , where the weight of each action category is obtained by dividing the median of all action category frequencies by the frequency of each action category;

[0109] In order to further improve the prediction quality, existing methods usually use additional smoothing losses to reduce this over-segmentation error, such as using the truncated mean square error T-MSE on the inter-frame log probability, which is expressed as follows:

[0110]

[0111]

[0112] Δ t，c =|logy t，c -logy t-1，c |;

[0113] However, T-MSE penalizes all frames in a video to smooth the transitions between action probabilities. This results in penalizing frames with actual action transitions. In other words, when T-MSE is applied to a video, it penalizes all frames regardless of whether the current frame is a boundary. This smoothes actual boundaries in the video and can lead to blurred boundaries. To address this issue, we propose Gaussian similarity-weighted T-MSE (GS-T-MSE) as a smoothing loss function.

[0114] Therefore, in this embodiment, Gaussian similarity weighted T-MSE (GS-T-MSE) is proposed as a smoothing loss function, that is, the smoothing loss when smoothing the frame confidence score and the fusion classification score in the cascade stage by the audio-visual bimodal boundary smoothing operator. Its expression is as follows:

[0115]

[0116]

[0117] Δ t，c =|logy t，c -logy t-1，c |;

[0118] in, is the smoothing loss; C n is the total number of categories; σ is the standard deviation; x t is the frame confidence score at time t; is the square of the logarithmic difference between the confidence scores of the two frames before and after; Δ t，c is the logarithmic difference of the confidence scores of the two frames before and after; τ is the threshold.

[0119] In this embodiment, the final loss function of the cascade stage is expressed as:

[0120]

[0121] Among them, λ is a model hyperparameter that determines the contribution of different losses; is the final loss function of the cascade stage;

[0122] The final loss function of the fusion stage is expressed as:

[0123]

[0124] in, is the final loss function in the fusion stage.

[0125] In this embodiment, in order to train the complete model, the sum of the losses of all cascade stages and fusion stages is minimized. The expression for minimizing the sum of the losses is:

[0126]

[0127] in, is the sum of the minimized losses of all cascade stages and fusion stages.

[0128] In this embodiment, a binary logistic regression loss function is designed for audiovisual bimodal boundary regression, and its expression is as follows:

[0129]

[0130] where y t is the true probability of time frame t; p t is the probability of the action boundary at time frame t. p Positive samples (i.e., all frames except the background) are weighted because the number of frames that are action boundaries is much smaller than the number of other frames. The ratio of positive data points in the entire training data is calculated and its inverse is used as the weight. Finally, the loss of each boundary prediction in the audio-visual bimodal boundary regression module is averaged to obtain the loss function of the audio-visual bimodal boundary regression, which is expressed as follows:

[0131]

[0132] Where N is the number of prediction stages; is the loss function for audiovisual bimodal boundary regression; is the loss value of boundary regression at each stage.

[0133] Experiment and result analysis

[0134] For the EPIC-KITCHENS dataset, the audio and video features of the dataset were extracted using the audio and video feature extraction method proposed in this patent to verify the effectiveness of the video time-domain action segmentation method based on multi-stage audio and video dual-modal feature fusion proposed in this patent.

[0135] Datasets and evaluation metrics

[0136] We use audio and video features extracted from the EPIC-KITCHENS dataset. Table 1 lists the number of video features, which are 633 with a feature dimension of 1024, and the number of audio features, which are 633 with a feature dimension of 128. Evaluation metrics include frame-level accuracy (Acc), segment-wise edit distance (Edit), and segment-wise F1 scores at overlap thresholds of 10%, 25%, and 50%, denoted by F1@{10, 25, 50}.

[0137] Experimental parameter settings

[0138] A multi-stage audiovisual bimodal cascade network with three cascade stages and one fusion stage is used. Each stage contains 10 dilated convolutional layers, where the dilation rate of the dilated convolutions doubles with the number of layers. Dropout layers are used after each layer to prevent overfitting. The number of filters in all layers of the cascade network is set to 64, and the filter size is 3. The audiovisual bimodal boundary regression module adopts a multi-stage structure with an initial prediction stage and three optimization stages. Each stage consists of a temporal convolution with a kernel size of 1 and 64 filters, 10 dilated residual convolutions, and another temporal convolution to reduce the feature dimensionality to the number of action classes. The dilation rate of each dilated convolution doubles with the number of layers. The loss function is set to τ = 4 and λ = 0.15. In all experiments, the Adam optimizer, commonly used in deep learning, is used with a learning rate of 0.0005.

[0139] Ablation experiments

[0140] (1) The impact of the number of stages in the audiovisual bimodal boundary regression module

[0141] The audio-visual bimodal boundary regression module needs to consider the impact of the number of stages on the video segmentation effect. The impact of the number of stages of the audio-visual bimodal boundary regression module is compared on the 50Salads dataset.

[0142] The effect of the number of stages of the audiovisual bimodal boundary regression module on the EPIC-KITCHENS dataset is shown in Table 3. Models using different numbers of stages of the audiovisual bimodal boundary regression module all achieved relatively good frame-level accuracy Acc scores. However, observing the segmentation edit distance Edit and segmentation F1 scores of these models, the F1 score of the single-stage model is low, which may be because the number of stages is too small to capture enough boundary information, resulting in a large number of over-segmentation errors. From the improvement in segmentation F1 scores after the number of stages increases, it can be seen that the multi-stage architecture helps capture boundary information. However, after adding the fourth stage, it can be seen that the performance begins to decline, which may be due to the overfitting problem caused by the increase in the number of parameters. Therefore, in subsequent experiments, an audiovisual bimodal boundary regression module with three stages is used.

[0143] Table 3 The impact of the number of stages of the audiovisual bimodal boundary regression module on the EPIC-KITCHENS dataset

[0144]

[0145] (2) The role of audio features in video action segmentation

[0146] The proposed video temporal action segmentation network model introduces audio feature information into the field of video action segmentation technology for the first time. The following compares the role of audio features in video action segmentation.

[0147] Table 4 describes the variants of the video temporal action segmentation network model MS-AVBAN. MS-BAN represents a multi-stage audio-visual multimodal cascade network for the video modality. MS-AVBAN (fusion) indicates that audio features participate in both boundary regression and frame-level action classification. MS-AVBAN (classification) indicates that audio features participate only in frame-level action classification and do not participate in boundary regression. MS-AVBAN (boundary) indicates that audio features participate only in boundary regression and do not participate in frame-level action classification.

[0148] Table 4 Description of the video temporal action segmentation network model MS-AVBAN variant

[0149]

[0150] Table 5 shows the comparison results of the variants of the video temporal action segmentation network model on the EPIC-KITCHENS dataset, which compares the optimization effect of the audiovisual bimodal boundary regression module on the multi-stage audiovisual multimodal cascade network and the influence of audio features on the frame-by-frame prediction results of the multi-stage audiovisual multimodal cascade network.

[0151] MS-AVBAN (fusion) achieved the best results, improving F1@{10,25,50}, segmented edit distance (Edit), and frame-level accuracy (Acc) by 2.5%, 2.4%, 4.3%, 2.3%, and 3.5%, respectively, compared to MS-BAN. This demonstrates that the combined use of audio features in boundary regression and frame-level action classification effectively improves the network's action segmentation results. MS-AVBAN (fusion), MS-AVBAN (classification), and MS-AVBAN (boundary) all achieved better segmented edit distance (Edit) and segmented F1 scores than MS-BAN, demonstrating that incorporating audio features into action segmentation can effectively balance the issues of over-segmentation and blurred boundaries.

[0152] Table 5 Comparison results of variants of video temporal action segmentation network models on the EPIC-KITCHENS dataset

[0153]

[0154] Comparative experiments on the video temporal action segmentation network model MS-AVBAN and the most advanced methods

[0155] On the extracted audio-visual bimodal feature dataset, the video temporal action segmentation network model is compared with the state-of-the-art methods (MS-TCN, MS-TCN++, SSTDA+HASR, DTGRM, MS-TCN+HASR, ASRF, BCN and SSTDA). The experimental results are shown in Table 6.

[0156] To reasonably evaluate the effectiveness of the model, we uniformly used extracted features from the EPIC-KITCHENS dataset to experiment with existing methods. This is because the most advanced models in the field of video temporal action segmentation all use video features for action segmentation, while the video temporal action segmentation network model proposed in this application is the first in the field to incorporate audio features. Therefore, when experimenting with other methods, we only used video features from the EPIC-KITCHENS dataset.

[0157] Table 6 Comparison results with the most advanced methods on the EPIC-KITCHENS dataset

[0158]

[0159] The video temporal action segmentation network model achieved state-of-the-art performance in the evaluation metrics of frame-level accuracy (Acc), segmented edit distance (Edit), and segmented F1 score. It achieved a 3.3% improvement in frame-level accuracy (Acc), a 2.2% improvement in segmented edit distance (Edit), and 2.3%, 2.4%, and 4.3% improvements in F1@{10,25,50}, respectively. The proposed multi-stage audiovisual multimodal cascade network (MS-BAN) also effectively captured video information, ranking among the top methods. However, its performance was far inferior to the video temporal action segmentation network model, demonstrating the power of audio feature information in resolving over-segmentation and blurred boundaries.

[0160] In order to demonstrate the real segmentation results of the video temporal action segmentation network model, two representative videos were selected from the EPIC-KITCHENS dataset for example analysis. Taking the proposed multi-stage audio-visual multimodal cascade network MS-BAN as a reference, the qualitative results of the video temporal action segmentation network model MS-AVBAN and the multi-stage audio-visual multimodal cascade network MS-BAN are shown in the figure. Figure 5 As shown in the figure, GT represents the true label, and each color block represents an action segment. From the qualitative results, although MS-BAN has achieved very good results, there are still some problems such as blurred boundaries and over-segmentation.

[0161] Blurred boundaries: Figure 5As shown in (a), MS-BAN mistakenly merges the second and third actions into a single action segment. This is likely because, to mitigate the over-segmentation problem, MS-BAN uses Gaussian similarity weighted T-MSE in its model, failing to fully distinguish boundaries and resulting in severe boundary blurring. MS-AVBAN, on the other hand, captures the waveform characteristics of the audio, avoiding smoothing boundary information and effectively alleviating the boundary blurring problem. Therefore, MS-AVBAN correctly segmented the second and third actions.

[0162] Over-segmentation: e.g. Figure 5 As shown in (b), MS-BAN produces some over-segmentation errors, predicting other actions in one action segment. The reason for MS-BAN's poor performance is that the self-attention boundary smoothing operator is not sufficient to solve the over-segmentation problem. MS-AVBAN, on the other hand, introduces audiovisual bimodal boundary information into a multi-stage cascade network, optimizing frame-level action prediction at each stage. This results in more accurate prediction results and effectively solves the over-segmentation problem.

[0163] In summary, the multi-stage audiovisual bimodal boundary-aware video temporal action segmentation model flexibly uses cascade networks of different stages for processing based on the difficulty of classifying actions using audio and video features. Low-stage cascade networks process easily classified audio and video, while high-stage cascade networks process difficult-to-classify audio and video. Simultaneously, the natural waveform features of the audio are utilized to regress the probability of action boundaries to obtain an audiovisual bimodal boundary smoothing operator, optimizing the prediction results at each stage. Therefore, the video temporal action segmentation model proposed in this invention can effectively alleviate the problem of blurred video segmentation boundaries and improve the accuracy of video segmentation.

[0164] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A video temporal action segmentation method based on audio and video dual-modal feature fusion, characterized by: The following steps are involved: S1: Extract video and audio features of multimedia data; S2: Establish a video temporal action segmentation network model and output a video temporal action segmentation classification result based on the video temporal action segmentation network model. The video temporal action segmentation network model includes a multi-stage audio-visual bimodal cascade network and an audio-visual bimodal boundary regression module. The specific steps include: S2-1: Integrate the audio features of multimedia data into the video features to obtain the fused audio-visual bimodal features, which are then input into a multi-stage audio-visual multimodal cascade network. S2-2: Based on audio and video features, an audio-visual bimodal boundary smoothing operator is generated through the audio-visual bimodal boundary regression module. The audio-visual bimodal boundary smoothing operator is used to smooth the frame confidence scores of each cascade stage in the multi-stage audio-visual multimodal cascade network and generate self-attention weights. S2-3: The frame confidence scores of each cascade stage in the multi-stage audio-visual multimodal cascade network are weighted and summed according to the self-attention weight matrix, and input into the fusion stage to obtain the fusion classification score. The video temporal action segmentation classification result is output based on the fusion classification score; The expression for smoothing the frame confidence score of each cascade stage in the multi-stage audiovisual multimodal cascade network by the audiovisual bimodal boundary smoothing operator in step S2-2 is: b′=(b video +βb audio ) / 2 in, is the frame confidence score predicted by the network after adding the smoothing operator; is the frame confidence score; s is the direction of pooling; β is the model hyperparameter that determines the contribution of different features; L is the length, and the length of the pooling window is 2L+1; is the frame confidence score of the frame in the pooling window; α is the decay rate of the self-attention weight; j is the number of elements taken in each direction in the pooling window; b′ t+s·j is the action boundary probability within the pooling window; att is the self-attention adjustment unit; b′ is the fusion boundary probability; b video is the video boundary probability; b audio is the audio boundary probability; The expression of the self-attention weight in step S2-2 is: in, is the self-attention weight of each cascade stage in the multi-stage audio-visual multimodal cascade network; e is a natural constant; ρ is the set frame confidence score threshold; is the frame confidence score at time t and stage j'.

2. The video temporal action segmentation method according to claim 1, characterized in that: The expression of the fusion classification score in step S2-3 is: in, is the fusion classification score.

3. The video temporal action segmentation method according to claim 2, characterized in that: The step S2-3 further includes the following steps: smoothing the fusion classification scores by using an audiovisual bimodal boundary smoothing operator, which is expressed as follows: b′=(b video +βb audio ) / 2 in, It is the smoothed fusion classification score predicted by the network after adding the smoothing operator.

4. The video temporal action segmentation method according to claim 3, characterized in that: In order to ensure that the model training tends to stable convergence, a loss function of the multi-stage audio-visual bimodal cascade network structure is designed. The loss function includes the loss distribution on the frame in the cascade stage, and the expression is as follows: in, is the loss distribution on the frame in the cascade stage; y t,c is the predicted probability of category c at time t; t is a certain time; And the classification loss on the frame in the fusion stage is expressed as follows: in, is the classification loss; T is the total length of multimedia data.

5. The video temporal action segmentation method according to claim 4, characterized in that: The loss function of the multi-stage audiovisual bimodal cascade network structure also includes a smoothing loss, which is the smoothing loss when smoothing the frame confidence score and the fusion classification score of the cascade stage through the audiovisual bimodal boundary smoothing operator. Its expression is as follows: in, is the smoothing loss; C n is the total number of categories; σ is the standard deviation; x t is the frame confidence score at time t; is the square of the logarithmic difference between the confidence scores of the two frames before and after; Δ t,c is the logarithmic difference of the confidence scores of the two frames before and after; τ is the threshold.

6. The video temporal action segmentation method according to claim 5, characterized in that: The final loss function of the cascade stage is expressed as: Among them, λ is a model hyperparameter that determines the contribution of different losses; is the final loss function of the cascade stage; The final loss function of the fusion stage is expressed as: in, is the final loss function in the fusion stage.

7. The video temporal action segmentation method according to claim 6, characterized in that: The expression for minimizing the sum of losses of all cascade stages and fusion stages is: in, is the sum of the minimized losses of all cascade stages and fusion stages.

8. The video temporal action segmentation method according to claim 7, characterized in that: It also includes the loss function of audiovisual bimodal boundary regression, which is expressed as follows: Where N is the number of prediction stages; is the loss function for audiovisual bimodal boundary regression; is the cross entropy loss at each stage of boundary regression.

Images